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Abstract 

Ear  too  often  decision  makers  select  concepts  based  on  insufficient  data,  resulting 
in  projects  that  are  over-budget,  over-schedule,  and  not  what  the  customer  wants. 
Research  efforts  have  proposed  a  stage-gated  concept  maturity  framework  as  a  tool  to 
assess  and  increase  the  maturity  of  concepts.  This  research  uses  multiple  validation 
techniques  to  demonstrate  the  value  this  framework  can  provide.  Interviews  with 
acquisition  professionals  capture  qualitative  and  quantitative  data  on  the  utility  of  the 
elements  of  the  framework  and  the  acquisition  process.  This  research  also  applies  the 
framework  to  a  current  acquisition  program  to  determine  if  it  can  be  broadly  applied  for 
different  types  of  developments.  Eastly,  this  research  looks  to  current  acquisition  policy 
and  guidance  to  see  if  there  is  support  for  the  maturity  elements  of  the  framework. 

The  results  of  this  study  led  the  research  team  to  accept  the  framework  as  a  useful 
guide  and  approach  to  assessing  a  concept’s  maturity.  The  majority  of  responses  were 
favorable  towards  the  activities  recommended  in  the  framework.  The  researchers  were 
able  to  apply  the  framework  in  real-time  to  a  concept  in  early  development  to  the  benefit 
of  the  sponsoring  organization.  The  results  of  this  study  have  also  led  to  the  formation  of 
themes,  best-practices,  and  lessons-leamed  concerning  early  concept  development.  The 
results  affirm  that  when  developing  a  concept  people  make  the  difference,  more  resources 
up-front  are  needed  to  fully  understand  a  concept,  and  developers  should  avoid 
constraining  the  trade-space  by  pre-supposing  a  solution. 
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I.  Introduction 


General  Issue 

It  has  been  demonstrated  often  that  when  time  and  effort  is  invested  early  on  in 
a  system’s  development  that  the  return  on  investment  is  significant  (Government 
Accountability  Office,  2008).  Recently  there  have  been  many  ongoing  efforts  to  help 
quantify  this  early  systems  engineering  process.  These  efforts  have  the  goal  to  create  a 
disciplined  and  repeatable  process.  Such  a  process  will  help  guide  future  efforts 
towards  constructive  activities  that  will  add  value  during  the  development. 

Problem  Statement 

According  to  (Hughes,  2010)  a  mature  concept  is  one  that  contains  the  right 
amount  of  information  at  the  right  development  stage.  Even  if  a  concept  is  in  its 
infancy  it  may  be  very  mature  if  the  concept’s  capabilities  and  limitations  are  fully 
understood.  Conversely  concepts  that  have  existed  for  a  very  long  time  may  be 
immature  if  the  capabilities  and  limitations  are  not  well  understood  or  documented. 

Determining  if  a  concept  is  mature  early  on  in  system  development  can  be 
difficult  to  assess.  Developers  can  use  systems  engineering  tools  to  help  to  uncover 
the  “unknown  unknowns”  that  can  plague  a  systems  development.  These  problems  are 
often  caused  by  unclear,  undefined,  or  unattainable  goals  during  systems  development. 
The  concept  maturity  model  developed  by  Hughes  attempts  to  quantify  the  risks  and 
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unknowns  associated  with  each  proposed  concept.  The  more  thoroughly  the  concept  is 
understood  and  documented,  the  more  informed  a  decision  can  be  made.  This  more 
informed  decision  will  help  mitigate  the  risks  inherent  to  development  of  the  system. 
Research  Objectives/Questions/Hypotheses 

Previous  research  has  proposed  a  “Concept  Maturity  Assessment  Framework” 
method  in  which  decision  makers  can  determine  the  maturity  of  concepts  as  they 
progress  through  the  acquisition  process.  The  framework  as  developed  by  Hughes 
(2010)  with  collaboration  from  Barker  (2010)  will  be  referred  to  as  the  Hughes’ 
Framework  throughout  this  study.  This  thesis  proposal  will  take  the  Hughes’ 
Framework  and  first  demonstrate  its  application.  Secondly  this  thesis  proposal  will 
attempt  to  determine  the  utility  and  added- value  of  the  framework  to  the  decision 
maker.  In  other  words,  this  thesis  seeks  to  quantify  and  qualify  the  added-value  of 
applying  the  framework  assessment,  and  give  a  recommendation  for  or  against  future 
use  during  the  acquisition  process.  The  questions  to  be  addressed  are: 

1.  Is  the  framework  a  valid  guide,  that  can  be  used  to  assess  concept 
maturity? 

2.  Can  the  framework  be  applied  in  real-time  during  concept 
development? 

3.  What  is  the  added-value  to  the  decision  maker  in  applying  the 
framework? 

4.  Does  the  framework  help  reduce  or  mitigate  the  risk  associated  with 
concept  maturity  for  the  decision  maker? 

5.  Should  the  framework  be  recommended  for  use  during  the  acquisition 
process? 
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Methodology 

To  answer  the  research  questions  above,  a  structured  three-level  approach  will 
be  used  to  show  how  the  framework  can  be  applied  and  then  assess  the  value-added  to 
the  decision  maker.  The  methodology  for  this  research  will  be  focused  on  reasoning 
derived  from  interviews  with  relevant  personnel,  a  real-time  application  of  the 
framework,  and  a  comparison  to  more  commonly  accepted  sources. 

The  primary  method  of  this  validation  effort  uses  a  structured  interview 
approach  that  seeks  to  understand  the  perceptions  of  relevant  personnel  related  to  the 
framework.  The  researchers  will  gather  the  opinions  and  recommendations  of  relevant 
experts,  users,  and  practitioners  with  current  or  recent  experience  related  to  early 
systems  development.  The  results  from  these  interviews  will  form  the  foundation  to 
help  form  an  unbiased  evaluation  on  the  usefulness  of  the  framework. 

Demonstrating  the  framework’s  application  will  be  focused  on  applying  the 
framework  to  a  weapon  system  that  will  destroy  hard-to-defeat  targets.  This  weapon 
system  is  currently  in  the  concept  development  phase.  Two  separate  defense 
contractors  are  competing  for  the  opportunity  to  develop  this  weapon  system.  The 
authors  will  use  the  framework  to  assess  the  maturity  of  the  separate  concepts  from 
each  contractor  and  provide  individual  assessments  to  the  responsible  sponsoring 
military  organization.  This  assessment  will  serve  to  demonstrate  how  the  framework 
can  be  applied  to  assess  concept  maturity.  The  researchers  will  also  help  develop 
some  of  the  information  products  as  recommended  by  the  assessment. 

This  assessment  will  not  solely  focus  on  the  work  of  the  competing  contractors. 
The  framework  will  be  used  to  assess  the  work  done  by  the  government  in  defining  the 
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need.  Did  the  government  adequately  define  the  needs,  objectives,  tasks  and/or 
measures  the  concepts  will  be  judged  against?  Were  the  expectations  clearly  defined? 
These  are  just  two  of  the  questions  this  assessment  will  attempt  to  address. 

The  final  step  will  be  a  review  of  accepted  sources  that  help  confirm  or  negate 
the  conclusions  drawn  while  using  of  the  framework.  This  approach  is  not  meant  to 
stand-alone  as  a  method  for  validation,  but  serves  to  support  the  findings,  if  applicable, 
derived  from  the  structured  interviews.  The  research  team  will  evaluate  approved 
policy  and  regulations  as  they  relate  to  the  framework  as  well  as  guidance  and  best- 
practices  from  practitioners  to  complete  this  assessment. 

Assumptions/Limitations 

The  Hughes’  Framework  has  been  defined  in  a  previous  thesis  with  three 
staged  gates  to  assess  concept  maturity  based  on  its  current  phase  in  the  system  / 
product  development  process.  These  gates  can  be  aligned  to  commercial  and 
Department  of  Defense  (DoD)  acquisition  processes.  As  the  concept  being  considered 
for  development  progresses  through  the  phases  of  the  acquisition  process,  its  maturity 
should  also  progress  and  be  judged.  This  thesis  assumes  that  the  framework  is  ready 
and  complete  for  application  and  no  further  changes  will  be  made  prior  to  validation. 
The  authors  of  this  thesis  have  no  pre-conceived  opinion  as  to  the  usefulness  of  the 
framework. 

Even  though  the  framework  developed  by  Hughes  can  be  applied  to  both 
commercial  and  DOD  developments  the  research  in  this  thesis  will  focus  on  the  DOD 
relationships.  The  following  section  describes  briefly  the  stage-gated  process  of  the 
framework. 
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Gate  1:  Opportunity  Identification  takes  the  approved  Initial  Capabilities 
Document  (ICD)  and  assesses  the  proposed  concepts  on  their  ability  to  fill  the 
capability  gap.  If  analysis  determines  that  a  new  material  solution  is  necessary  the 
concept  maturity  assessment  requires  the  consideration  of  various  aspects  of  the 
program  such  as  architecture  products  found  in  the  Department  of  Defense 
Architecture  Framework  (DoDAF),  a  CONOPS,  AV-1,  OV-5a/b,  OV6-a,  OV-4,  OV- 
2,  CV-2,  CV-4,  CV-6,  and  so  on.  These  architecture  documents  help  characterize  all 
aspects  of  the  proposed  concept  and  how  it  will  interact  with  everything  else.  These 
documents,  if  thorough  and  complete,  will  reduce  the  risk  of  “unknown  unknowns” 
and  provide  a  maturity  assessment. 

Gate  2:  Concept  Screening  identifies  and  narrows  down  the  possible  solution 
concepts  that  should  be  pursued  further  through  this  phase.  This  gate  is  aligned  with 
the  Material  Development  Decision  (MDD)  in  the  DoD  acquisition  process.  The 
information  required  at  this  gate  is  the  updated  architectural  documents  from  the 
previous  gate  as  well  as  architectural  documents  that  technically  define  the  concept. 

As  the  concept  has  been  refined,  its  functionality,  interactions,  and  cost/schedule 
projections  must  all  be  updated  so  that  any  changes  that  have  been  made  are 
thoroughly  understood  and  their  impact  is  fully  assessed.  Having  the  most  up  to  date 
information  is  vital  at  this  step  so  that  only  the  best  concepts  are  allowed  to  proceed. 

Gate  3:  Concept  Selection  identifies  and  narrows  yet  again  the  concept(s)  that 
should  be  pursued  further  through  an  analysis  of  alternatives.  This  gate  is  aligned  with 
Milestone  A  (MSA)  in  the  DoD  acquisition  process.  The  information  required  at  this 
gate  is  an  additional  update  to  the  previous  architecture  documents.  Similar  to  Gate  2, 
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as  the  concept  has  been  refined,  its  functionality,  interactions,  and  cost/schedule 
projections  must  all  be  updated  so  that  any  changes  that  have  been  made  are 
thoroughly  understood  and  their  impact  is  fully  assessed. 

Implications 

If  the  Hughes’  Framework  proves  to  be  a  valid  process  and  provides  value  to 
the  acquisition  process  by  better  defining  and  characterizing  the  risks  associated  with 
the  maturity  /  immaturity  of  proposed  concepts  then  future  efforts  should  be 
undertaken  to  integrate  this  framework  into  the  DoD  acquisition  process.  Some 
potential  vehicles  to  help  integrate  this  framework  would  be  the  Early  Systems 
Engineering  and  Concept  Characterization  and  Technical  Description  (CCTD)  guides, 
which  have  already  laid  the  foundation  to  increase  the  level  of  systems  engineering 
efforts  early  on  in  the  acquisition  process.  Integrating  the  required  documentation  for 
the  stage  gated  concept  maturity  model  into  the  Early  Systems  Engineering  and  CCTD 
guides  could  help  prevent  unexpected  surprises  from  arising  by  thoroughly 
documenting  and  understanding  key  considerations  of  a  concepts  maturity. 

This  chapter  has  identified  the  problem  which  the  research  will  address.  The 
objectives  of  the  research  have  been  laid  out  and  the  questions  this  research  seeks  to 
answer  have  been  proposed.  The  implications  of  this  research  if  successful  have  been 
discussed  along  with  the  limitations  and  assumptions.  Chapter  two  goes  into  the 
background  information  that  went  into  the  formulation  of  the  framework  developed  by 
Hughes  as  well  as  a  quick  explanation  of  the  existing  DoD  acquisition  process  and 
how  the  framework  relates  to  it.  Chapter  three  discusses  in  detail  the  methodology  and 
research  approach  used  to  validate  the  Hughes’  Eramework.  Chapter  four  starts  with 
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the  analysis  of  the  structured  interviews  that  were  conducted  and  then  discusses  the 
practical  application  of  the  framework  that  the  research  team  performed  and  ends  with 
analysis  of  current  policy.  Chapter  five  will  discuss  the  conclusions  drawn  from  the 
analysis,  answer  the  research  questions,  and  present  themes  and  lessons-learned. 
Chapter  six  includes  recommendations  the  research  team  has  concerning  the  Hughes’ 
Framework  and  proposes  further  research. 
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II.  Background 


Chapter  Overview 

This  chapter  discusses  several  important  components  associated  with 
validating  the  Hughes’  Framework.  Discussed  first  is  the  DoD  development  process 
with  its  decision  gates  and  how  concept  maturity  is  a  factor  in  early  milestone 
decisions.  The  Hughes’  Framework  will  then  be  explained  in  detail.  This  detailed 
understanding  of  the  different  maturity  elements  called  out  in  the  Hughes’  Framework 
will  be  the  basis  of  the  validation  research.  The  different  maturity  elements  of  the 
framework  will  be  subjected  to  validation  techniques  to  determine  their  utility. 
Understanding  DoD 

DoD  Acquisition  often  finds  itself  in  the  precarious  position  of  having  to 
satisfy  conflicting  stakeholders’  needs.  The  user  communities  are  starving  for  new 
and  improved  systems  and  technologies,  and  they  want  them  yesterday.  The  DoD 
attempts  to  meet  the  user  needs  by  developing  weapon  systems  faster.  At  the  same 
time,  congressional  oversight  expects  the  DoD  to  build  systems  on  a  strict  budget,  with 
heavy  requirements  for  documentation  and  reporting  which  often  leads  to  a  direct 
conflict  with  the  users  demand  for  increased  performance  and  faster  schedules.  As  a 
result  of  these  pressures,  the  DoD  often  enters  into  contracts  with  defense  contractors 
before  requirements  for  the  needed  systems  have  been  properly  assessed  and  analyzed. 
The  contractors,  in  an  attempt  to  maintain  the  schedule  while  staying  within  a  tight, 
closely  monitored  budget,  often  omit  essential  early  planning  steps,  the  very  steps  that 
would  likely  help  them  to  succeed  in  the  “long-term”  (Government  Accountability 
Office,  2008).  Further,  stakeholders  decide  to  change  or  add  requirements  to  meet 
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more  current  needs,  which  only  cause  additional  delays  and  incurs  additional  costs,  the 
very  two  things  they  were  trying  to  avoid  in  the  first  place.  These  redesigns  must 
navigate  through  entire  supply  chains,  affecting  many  or  most  of  the  development 
team  members  (Government  Accountability  Office,  2008). 

As  previously  mentioned,  the  process  to  initiate  a  weapons  system  within  the 
DoD  usually  begins  with  the  identification  of  an  operational  need  found  in  a  mission 
area.  The  DoD  attempts  to  identify  mature  technologies  within  the  Federal 
Government  and  commercial  industry  that  would  potentially  satisfy  the  operational 
need  (PMBOK  Guide,  2008).  A  Capability-Based  Assessment  (CBA)  is  also  done  in 
parallel  to  determine  if  the  need  can  be  satisfied  by  a  non-materiel  solution.  If  it  is 
determined  that  a  materiel  solution  is  required,  an  Initial  Capabilities  Document  (ICD) 
is  then  produced,  which  depicts  the  operational  deficiency,  as  well  as  an  opportunity  to 
provide  a  new  capability  (PMBOK  Guide,  2008).  In  an  effort  to  produce  the  ICD, 
needed  capabilities  are  analyzed  and  potential  nascent  concepts  are  birthed.  These 
early  concepts  are  considered  the  seed  corns  of  future  systems  and  are  expected  to 
address  capability  shortfalls  or  to  exploit  new  capabilities  provided  by  new 
technologies  (SAF/AQ,  2009).  The  definition  of  a  concept  is  widely  debated  but  shall 
be  defined  in  this  work  as  “a  solution  that  meets  the  needs  of  the  customers  and  users 
and  identifies  the  resources  required  to  develop  the  solution”  (Hughes,  2010). 

Early  Decision  Points 

It  is  important  to  understand  the  considerations  and  discriminators  for  each 
progressive  investment  decision  in  the  concept  development  process.  This  validation 
effort  focuses  on  the  three  decision  gates  discussed  in  the  Hughes’  Framework  as  well 
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as  the  information  maturity  elements  used  to  prepare  for  those  decision  gates  (Hughes, 
2010).  The  first  gate,  Opportunity  Identification,  occurs  when  the  Joint  Requirements 
Oversight  Council  (JROC)  approves  and  validates  an  ICD  (Chairman  of  the  Joint 
Chiefs  of  Staff,  2009).  The  ICD  contains  requirements  originating  from  user  needs. 
The  analysis  conducted  by  the  user  to  develop  an  ICD  should  identify  the  shortfall  in 
military  capability  and  determine  that  a  new  materiel  product  is  required  to  meet  the 
need.  The  JROC  must  determine  if  this  development  opportunity  is  important  enough 
for  the  allocation  of  resources. 

If  the  JROC  determines  that  the  ICD  is  complete  and  that  it  identifies  a  valid 
need,  the  next  gate  of  the  Hughes’  Framework,  Concept  Screening,  corresponds  to  a 
similar  gate  in  the  DoD  5000  Framework,  Materiel  Development  Decision  (MDD). 
The  primary  purpose  of  the  MDD  is  to  act  as  the  official  entry  gate  to  the  materiel 
development  process.  It  also  acts  as  a  filter  to  prevent  the  concepts  that  are  unfeasible 
for  development  or  that  do  not  meet  the  need  identified  in  the  ICD  from  progressing 
any  further  in  the  development  process.  A  concept  that  meets  the  criteria  and  contains 
adequate  information  will  undergo  deeper  analysis  in  the  Materiel  Solution  Analysis 
phase. 

The  third  screening  gate  of  the  Hughes’  Framework,  Concept  Selection,  occurs 
at  Milestone  A  (MS-A).  At  this  investment  decision,  a  concept  is  selected  based  upon 
a  set  of  criteria.  The  criteria  should  include  user  needs,  risk  associated  with 
development,  cost  to  manufacture,  operate  and  sustain  the  solution,  and  the  benefits 
the  development  brings  to  the  DoD.  A  concept  that  demonstrates  that  it  meets  the 
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criteria  and  is  selected  will,  assumedly,  have  resources  allocated  to  conduct 
preliminary  design  (Department  of  Defense,  2008). 

Departing  From  the  Process 

In  2008,  the  National  Research  Council  (NRC)  published  a  Secretariat  of  the 
Air  Force  Acquisition  Science,  Technology  and  Engineering  (SAF/AQR)- 
commissioned  study  entitled  “Pre-Milestone  A  and  Early-Phase  Systems  Engineering: 
A  Retrospective  Review  and  Benefits  for  Future  Air  Force  Systems  Acquisition.”  Key 
recommendations  of  this  report  found  that  in  many  cases  during  pre-MS-A  activities, 
“required  documents  were  completed  pro-forma  and  filed  away,  never  to  be  seen 
again,  or  for  which  required  steps  were  skipped  completely”  (NRC,  2008).  The 
problem  of  departing  from  the  structured  development  process  due  to  external  forces 
or  instability  surrounding  the  process  is  not  new.  This  structured  development  process 
is  great  once  the  need  and  gaps  are  fully  understood,  but  the  “fuzzy  front-end”  of 
development,  e.g.  truly  understanding  in  an  unbiased  manner  what  the  needs  and  gaps 
are,  requires  more  rigorous  definition.  Departures  from  the  process  are  often  driven  by 
unanticipated  problems  such  as  changing  customer  requirements  or  roadblocks  in  the 
approval  process  (Repenning,  Goncalves,  &  Black,  2001).  A  common  departure  from 
the  process  is  called  “firefighting,”  which  is  a  term  given  to  situations  where 
developer’s  resources  are  used  to  fix  unforeseen  problems  later  in  the  development 
cycle.  The  many  cost  and  schedule  risks  associated  with  firefighting  are  well 
documented  (Repenning,  Goncalves,  &  Black,  2001).  Since  early  phase  tasks  are 
often  skipped  in  order  to  save  resources  in  the  short-term,  they  routinely  receive  less 
attention  than  they  should  (Repenning,  Goncalves,  &  Black,  2001).  Developers  want 
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to  initiate  their  programs  as  early  as  possible,  even  if  that  means  bypassing  early 
systems  engineering  rigor.  The  GAO  reported  that  the  DoD  frequently  enters  into 
development  contracts  with  its  contractors  before  disciplined  system  engineering 
processes  have  been  completed.  This  practice  introduces  significant  cost  and  schedule 
risk  to  a  development  program  (Government  Accountability  Office,  2008). 

It  can  be  assumed  that  if  the  concept  selection  process  continues  to  function 
without  sufficient  rigor,  immature  concepts  will  continue  to  be  selected.  These  types 
of  pre-MS-A  decisions  have  a  disproportionately  large  impact  on  the  lifecycle  of  the 
program.  In  fact,  nearly  three-quarters  of  total  system  life  cycle  costs  are  influenced 
by  decisions  made  before  the  end  of  the  concept  refinement  phase  at  MS-A.  On  the 
other  hand,  about  three-quarters  of  life  cycle  funds  are  not  actually  spent  until  after  a 
production  decision  is  made  at  Milestone  C  (MS-C)  (NRC,  2008),  (see  Figure  1.), 
further  strengthening  the  argument  that  DoD  needs  to  place  more  emphasis  on  making 
good  early  decisions. 
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(Loren  &  Bullard,  2008) 
Figure  1  -  Impact  of  Early  Decision  Making 


Improving  Early  Decision  Making 

In  an  effort  to  improve  good  early  decision  making,  the  DoD  needs  a  process  to 

assess  the  maturity  of  concepts  as  they  are  screened  through  early  development.  This 

process  should  establish  a  baseline  by  which  all  concepts  are  equitably  judged  and 

reviewed.  In  regards  to  this  issue,  the  GAO  recommends: 

Taking  into  account  the  differences  between  commercial  product  development 
and  weapons  acquisitions,  we  have  recommended  that  DOD  adopt  a 
knowledge-based,  incremental  approach  to  developing  and  producing  weapon 
systems.  This  type  of  an  approach  requires  program  officials  to  demonstrate 
that  critical  technologies  are  mature,  product  designs  are  stable,  and  production 
processes  are  in  control  at  key  junctures  in  the  acquisition  process. 

(Government  Accountability  Office,  2007) 
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Defining  Concept  Maturity 

According  to  Hughes,  a  mature  concept  is  one  that  contains  the  right  amount  of 

information  at  the  right  development  stage  (Hughes,  2010).  In  a  stage  gated  process, 

the  right  information  is  determined  by  the  level  of  investment  associated  with 

developing  the  concept  for  the  next  stage.  This  information  helps  the  decision  maker 

determine  if  advancing  the  concept  is  a  good  investment.  The  pieces  of  information 

needed  by  the  decision  maker  at  these  investment  decision  points  can  be  described  as 

maturity  elements.  Thus,  the  collection  of  maturity  elements  at  the  appropriate 

development  stage  can  help  the  decision  maker  to  make  a  more  informed  decision.  A 

SAF/AQR  Guidance  Memo  dated  19  Dec  2008,  “Early  Systems  Engineering  Planning 

Documentation  and  Concept  Characterization  and  Technical  Description  (CCTD) 

Implementation”  contains  the  following  language: 

Better  assessments  of  concepts  for  the  use  of  disciplined  and  robust  technical 
planning  will  ultimately  reduce  the  risk  of  a  poorly  planned  concept  being 
selected  in  an  AoA,  and  represent  an  appropriate  approach  to  structure 
programs  for  success  and  acquisition  excellence.  (NRC,  2008) 

In  concept  development,  there  are  several  important  factors  in  determining  if 

the  concept  definition  provides  adequate  information.  The  decision  maker  is  faced 

with  the  task  of  asking  enough  questions  to  fully  comprehend  the  maturity  of 

individual  concepts.  The  decision  maker  uses  the  available  information  to  analyze 

operational  and  development  risks.  Is  it  possible  for  a  concept  to  be  considered  mature 

if  the  concept  includes  a  high  technology  development  risk,  as  long  as  the  risk  is  well- 

understood?  If  the  answer  is  yes,  this  would  imply  that  concept  maturity  is  more 

dependent  on  the  level  of  understanding  rather  than  the  technological  and  economical 
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feasibility.  Assuming  this  is  true,  if  the  concept  possesses  high  technology 
development  risk,  the  concept  can  continue  to  mature  if  it  also  possesses  a  supporting 
risk  management  plan. 

The  Hughes’  Framework 

The  framework  developed  by  Hughes  is  designed  to  act  as  a  benchmark  and 
common  language  for  those  evaluating  and  developing  material  concepts.  The 
framework  is  designed  around  three  major  decision  gates  in  the  front  end  of  the  DoD 
acquisition  process.  The  information  recommended  at  each  gate  is  intended  to 
characterize  the  needs  of  the  intended  users,  in  the  form  of  a  solution  to  meet  those 
needs,  or  the  resources  required  to  develop  the  solution.  The  decision  makers  must 
determine  if  the  concept  before  them  is  worthy  of  further  development  and  can  only 
make  that  determination  if  they  are  presented  with  the  right  type  and  amount  of 
information. 

The  information  that  is  developed  and  gathered  prior  to  each  decision  gate  is 
captured  through  a  structured  documentation  process.  Though  there  are  many 
different  ways  to  capture  the  data,  Hughes  uses  the  Department  of  Defense 
Architecture  Framework  (DoDAF)  views  as  a  way  to  present  much  of  the  information. 
Table  1,  below,  gives  a  description  of  the  DoDAF  2.0  views  that  can  be  used  to 
capture  the  information  needed  at  the  decision  gates. 
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Table  1  -  DoDAF  Views 


Description 

View 

Description 

View 

Legend;  All  View  -  AV  Operational  View  -  OV  Systems  View  -  SV 

Services  View  -  SvcV  Capability  View  -  CV  Standards  View  -  StdV 

Describes  a  Project's  Visions,  Goals, 
Objectives,  Plans,  Activities,  Events, 
Conditions,  Measures,  Effects 
(Outcomes),  and  produced  objects. 

AV-l 

A  mapping  of  system  functions 
(activities)  back  to  operational 
activities  (activities) 

SV-5a 

The  high-level  graphical/  textual 
description  of  the  operational  concept. 

OV-1 

A  mapping  of  systems  back  to 
capabilities  or  operational 
activities. 

SV-5b 

A  description  of  the  resource  flows 
exchanged  between  operational 
activities. 

OV-2 

The  emerging  technologies, 
software/hardware  products,  and 
skills  that  are  expected  to  be 
available  in  a  given  set  of 
timeframes  and  that  will  affect 
future  system  development. 

SV-9 

The  organizational  context,  role  or 
other  relationships  among 
organizations 

OV-4 

The  identification  of  services, 
service  items,  and  their 
interconnections 

SvcV-1 

The  capabilities  and  activities 
(operational  activities)  organized  in  a 
hierarchal  structure. 

OV-5a 

A  description  of  resource  flows 
between  services 

SvcV-2 

The  context  of  capabilities  and 
activities  (operational  activities)  and 
their  relation-ships  among  activities, 
inputs,  and  outputs 

OV-5b 

The  relationships  among  and 
between  systems  and  services  in  a 
given  architecture 

SvcV- 

3a 

One  of  three  models  used  to  describe 
operational  activity.  It  identifies 
business  rules  that  constrain  operations 

OV-6a 

The  functions  performed  by 
services  and  the  service  data  flows 
among  service  functions 

SvcV-4 

The  identification  of  systems,  system 
items,  and  their  interconnections 

SV-1 

A  mapping  of  services  back  to 
operational  activities 

SvcV-5 

A  description  of  resource  flows 
between  systems 

SV-2 

A  hierarchy  of  capabilities  which 
specifies  all  the  capabilities  that 
are  referenced  throughout  the 
architectural  descriptions 

CV-2 

The  relationships  among  systems  in  a 
given  Architectural  Description.  It  can 
be  designed  to  show  relationships  of 
interest,  (e.g.,  system-type  interfaces, 
planned  vs.  existing  interfaces). 

SV-3 

The  dependencies  between  planned 
capabilities  and  the  definition  of 
logical  groupings  of  capabilities 

CV-4 
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Description 

View 

Description 

View 

The  functions  (activities)  performed  by 
systems  and  the  system  data  flows 
among  system  functions  (activities). 

SV-4 

A  mapping  between  the 
capabilities  required  and  the 
operational  activities  that  those 
capabilities  support 

CV-6 

The  listing  of  standards  that  apply 
to  solution  elements 

StdV-1 

(Table  Adopted  from  Hughes,  2010) 


The  process  of  developing  a  concept  is  ordered  and  iterative.  Information  is 
developed  and  gathered  prior  to  each  decision  gate.  However,  this  phase  of  product 
development  is  often  very  “fuzzy”  so,  there  is  an  iterative  aspect  to  the  concept 
development  process.  With  each  new  phase  comes  a  greater  amount  of  required  detail, 
which  could  bring  new  revelations.  This  information  is  the  foundation  and  a  guide  for 
the  development  activities  following  the  gate.  This  ordered  activity  continues  through 
the  phases  preceding  the  three  gates  with  the  activities  of  each  subsequent  phase 
building  upon  what  had  been  accomplished,  see  Table  2.  Additionally,  circumstances 
may  change  during  the  course  of  development  that  may  alter  or  negate  the  work 
accomplished  in  previous  phases.  An  evaluation  should  be  conducted  to  determine  the 
impacts  of  any  changes  due  to  new  revelations  or  changing  circumstances. 
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Table  2  -  Architecture  Views  by  Gate 


Gate 

1 

2 

3 


All 

View 


1 


lA 


IT 


U 


Capabilities 

Views 


Gate 

2 

4 

6 

1 

lA 

lA 

lA 

2 

IT 

IT 

IT 

3 

U 

U 

U 

Operational  Views 


I 

2 

4 

5a 

5b 

6a 

lA 

lA 

lA 

lA 

lA 

lA 

IT 

IT 

IT 

IT 

IT 

IT 

U 

U 

U 

U 

U 

U 

Services  Views 

I 

2 

3a 

4a 

5 

lA 

lA 

lA 

lA 

lA 

IT 

IT 

IT 

IT 

IT 

U 

U 

U 

U 

U 

Systems  Views 


1  2  3  4  5a  5b  9 


lA 

lA 

lA 

lA 

lA 

lA 

IT 

IT 

IT 

IT 

IT 

IT 

IT 

U 

U 

U 

U 

U 

U 

U 

Standards  View 


1 


lA  -  Initial/  as-is 
IT  -  Initial/  to-be 
U  -  Update 


(Hughes,  2010) 


Concept  Evaluation  and  Selection  within  a  Stage-Gated  Process 

The  general  purpose  of  every  decision  gate  in  the  front  end  is  to  prevent  any 
concept  from  continuing  to  a  subsequent  development  phase  before  it  is  ready.  At  the 
end  of  a  development  phase,  the  development  team  needs  to  demonstrate  to  the 
decision  maker  that  the  concept  is  developed  enough  to  proceed  to  the  next  phase  and 
that  further  development  of  the  concept  will  benefit  the  organization  and  the  intended 
user.  The  elements  used  to  assess  and  mature  a  concept  should  define  the  level  of 
robust  early  planning  required  at  a  given  decision  point.  A  simple  way  to  understand 
the  appropriate  time  for  any  particular  element  is  to  relate  the  purpose  of  the  element 
to  the  specific  objective  of  the  decision  following  a  development  stage.  A  descriptive, 
stage-gated  process  based  upon  processes  used  by  the  Department  of  Defense  (DoD)  is 
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presented  here  (Figure  2).  The  framework  begins  with  the  concept  development 
phase. 


Concept  Evaluation  and  Selection  Within  a  Stage-Gated  Process 


Opportunity 

Concept 

Concept 

Identification 

Screening 

Selection 

R.  Hughes  2010 


Figure  2  -  Concept  Maturity  Framework 


The  first  gate,  Opportunity  Identification,  corresponds  to  when  the  Joint 
Requirements  Oversight  Council  (JROC)  approves  and  validates  an  Initial  Capabilities 
Document  (ICD)  (Chairman  of  the  Joint  Chiefs  of  Staff,  2009),  which  contains 
mission  requirements.  The  analysis  conducted  to  develop  an  ICD  should  identify  the 
shortfall  in  military  capability,  identify  the  user  needs,  and  determine  that  a  new 
development  product  is  required  to  meet  the  need.  The  JROC  must  determine  if  this 
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development  opportunity  that  was  identified  from  a  market  environmental  analysis  is 
adequately  important  for  the  allocation  of  resources. 

If  the  JROC  determines  that  the  ICD  is  complete  and  that  it  identifies  a  valid 
need,  the  next  gate,  Concept  Screening,  corresponds  to  the  Materiel  Development 
Decision  (MDD).  The  primary  purpose  of  the  MDD  is  to  act  as  the  official  entry  gate 
to  the  materiel  development  process.  It  also  acts  as  a  filter  to  prevent  the  concepts  that 
are  infeasible  for  development,  or  do  not  meet  the  need  identified  in  the  ICD,  from 
progressing  any  further  in  the  development  process.  Any  concept  that  the  decision 
makers  deem  sufficient  will  undergo  further  maturity  with  deeper  analysis  in  the 
Materiel  Solution  Analysis  phase. 

The  third  screening  gate.  Concept  Selection,  corresponds  to  Milestone  A.  At 
this  investment  decision,  a  concept  is  selected  based  upon  a  set  of  criteria.  The  criteria 
should  include  user  needs,  risk  associated  with  development,  cost  to  develop,  operate 
and  sustain  the  solution,  and  the  benefits  the  development  brings  to  the  organization. 

A  concept  that  demonstrates  that  it  meets  the  criteria  and  is  selected  will,  assumedly, 
have  resources  allocated  to  conduct  preliminary  design.  Later  gates  with  detailed 
design  and  fabrication/production  readiness  will  be  necessary,  but  substantial  policy 
and  guidance  for  these  later  gates  exist  for  the  DoD  (Department  of  Defense,  2008). 
The  information  collected  for  these  early  decision  gates  will  greatly  affect  and  support 
the  activities  of  the  following  development  phases. 

These  decision  gates  define  the  concept  maturity  milestones  that  are  used  to 
prevent  any  concept  from  progressing  to  a  phase  of  development  before  it  is  ready.  A 
concept  can  pass  through  a  decision  gate  if  the  products  for  the  current  phase  of  work 
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are  complete  and  if  the  decision  authority  determines  that  there  is  benefit  to  further 
development.  The  worthiness  of  a  concept  for  additional  development  is  dependent 
upon  contextual  issues  like  resource  constraints  and  political  climate,  in  addition  to  a 
concept’s  maturity.  A  development  team  has  no  control  over  the  contextual  issues  but 
it  can  ensure  the  proper  definition  and  analysis  associated  with  a  decision  gate  has 
been  completed.  In  an  effort  to  mature  a  concept  to  the  point  of  selection,  the 
practitioner  should  develop  and  accrue  a  robust  set  of  maturity  elements  that,  when 
combined,  will  provide  sufficient  information  to  the  final  decision  maker  and  will 
provide  the  foundation  upon  which  the  remainder  of  the  project  will  be  built. 

Information  Maturity  Elements  for  Gate  1. 

The  ICD  currently  has  no  requirements  for  architecture  products  beyond  a 
Concept  of  Operations  (CONOPS)  and  an  associated  Concept  Graphic  (OV-1). 
However,  there  are  several  architecture  maturity  elements  that  can  be  useful  in 
supporting  the  JROC’s  decision  at  this  gate,  and  they  can  be  developed  from  the 
documentation  and  information  currently  required  in  the  generation  of  the  Initial 
Capabilities  Document.  The  Joint  Ops  Concepts  and  CONOPS  (defined  in  AFPD  10- 
28  or  IEEE  Std  1362-1998)  can  be  used  to  determine  what  the  users  need  to  do  and 
how  they  expect  to  do  it.  A  well-defined  CONOPS  identifies  the  mission  area, 
timeframe,  assumptions  with  regards  to  projected  capabilities,  desired  effects  and  both 
the  necessary  and  supporting  capabilities  that  are  needed.  This  information,  as  well  as 
other  information  contained  within  a  CONOPS,  can  be  used  to  develop  an  overview  of 
the  system  architecture  products  that  will  characterize  the  information  associated  with 
the  desired  capability  (AV-1).  The  CONOPS  should  also  include  sequenced  actions 
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for  the  operational  and  support  scenarios  envisioned  by  the  user.  Using  tools  such  as 
Use  Case  modeling  or  traditional  functional  decomposition,  this  information  can  be 
captured  in  operational  activity  models  (e.g.,  OV-5a/b  in  the  DoDAF  (DoDAF  2.0, 
2009)),  which  show  what  must  be  done  and  gives  the  context  of  how  it  might  be  done. 
Any  known  rules  or  constraints  that  may  restrict  operations  should  be  captured  (e.g., 
OV-6a  in  DoDAF)  to  give  a  better  understanding  of  the  user  environment.  In  addition, 
the  identification  of  any  organizations  involved  in  the  activities  (OV-4)  and  resources 
that  flow  between  activities  (OV-2)  will  help  characterize  the  situation  for  the  design 
teams  in  future  phases.  Finally,  the  capabilities  associated  with  the  mission  (CV-2)  and 
how  those  capabilities  support  or  interact  with  other  operational  activities  (CV-6)  and 
with  each  other  (CV-4)  should  be  captured. 

Any  existing  systems  and/or  services  (e.g.  U.S.  Air  Force)  involved  with  the 
desired  capability  described  in  the  ICD  should  be  identified  (SV-1,  SvcVl)  and  their 
interactions  should  be  characterized.  If  further  definition  of  the  interaction  and  various 
systems  and  services  associated  with  the  concept  is  warranted,  resource  flows  and 
existing/planned  interfaces  can  be  identified  (SV-2,3,  SvcV-2,3).  In  order  to  determine 
gaps  between  needed  capability  as  defined  by  the  operational  activity  models,  e.g., 
OV-5,  and  current  system  capability,  system  and  services  functionality  descriptions 
can  be  developed  for  current  systems  (SV-4,  SvcV-4)  and  mapped  to  required 
operational  activities  using  traceability  matrices  (SV-5,  SvcV-5).  During  this  early 
needs  identification  phase,  these  systems  and  services  architecture  elements  would  be 
restricted  to  existing  systems/services  for  the  time  frame  of  interest,  and  would  contain 
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only  the  detail  necessary  to  identify  the  projected  operational  gaps  and  determine  the 
reason  for  the  gaps. 

It  should  be  noted  that  many  of  these  architecture  products  are  or  may  be 
required  at  later  gates  associated  with  the  DoD  acquisition  process,  but  early  collection 
of  the  information  and  definition  of  these  products  during  the  needs  identification 
phase  will  help  in  the  long-term  effort.  The  initial  development  of  these  architecture 
elements  before  the  first  decision  gate  serves  three  purposes.  First,  the  methodical 
development  of  the  elements  can  be  used  to  document  existing  capability,  clarify  any 
gaps  in  the  capability,  and  characterize  the  operational  risk  associated  with  the  gap. 
Second,  the  insight  gained  from  these  elements  can  aid  in  assessing  the  form  of 
solution  to  meet  the  capability  gap.  Lastly,  the  elements  serve  as  the  foundation  for 
future  development  phases.  Each  proposed  solution  will  be  designed  and  evaluated 
based  upon  requirements  developed  from  the  ICD  (Hughes,  2010). 

A  very  important  component  of  the  ICD  that  is  not  currently  being  adequately 
addressed  is  that  of  effectiveness  measures  (Sadauskas,  2008).  As  part  of  the  needs 
identification  process,  needed  capabilities  should  be  identified  in  terms  of  tasks, 
attributes  and  measures.  The  measures  at  this  level  are  best  described  as  mission  level 
Measures  of  Effectiveness  (MOE).  While  the  JCIDS  policy  has  always  required  the 
inclusion  of  MOE’s  in  the  ICD,  recent  reports  have  suggested  that  ICD’s  are  not 
adequately  addressing  how  the  operational  needs  are  to  be  quantified  for  subsequent 
evaluation  of  alternatives  (Sadauskas,  2008).  The  MOE’s  serve  a  similar  purpose  as 
the  initial  target  specifications  found  in  the  product  development  literature,  which  is  to 
guide  the  development  and  selection  of  potential  solutions.  Identification  of  MOE’s  is 
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critical  to  the  concept  maturation  process,  and  is  included  herein  as  one  of  the  maturity 
elements. 

One  final  maturity  element  that  is  already  required  by  CJCSI  3170.01G  (2009) 
and  should  be  developed  in  this  early  phase  is  an  operational  risk  assessment.  This 
risk  assessment  describes  the  risk  of  not  filling  the  operational  need.  In  DoD  terms, 
this  could  be  higher  projected  loss  rates,  greater  numbers  of  personnel  and  systems 
allocated  to  missions,  projected  lengthening  of  the  campaign  duration,  and/or 
increased  vulnerability  due  to  insufficient  deterrent  capabilities.  In  later  stages,  these 
operational  risks  will  be  weighed  against  the  cost  and  technical  risk  associated  with 
pursuing  a  materiel  solution. 

Information  Maturity  Elements  for  Gate  2. 

After  an  organization  decides  to  pursue  a  development  opportunity,  they 
should  identify  as  many  potential  solutions  as  possible  (CJCSM  3170.01D,  2009). 
These  ideas  should  be  developed,  combined  and  discarded  as  they  pass  through  a 
series  of  screens  so  that  only  a  few  of  the  best  ideas  remain  for  consideration 
(Wheelwright  &  Clark,  1992).  The  DoD  calls  this  screen  a  Materiel  Development 
Decision  and  uses  it  as  a  final  check  of  the  JROC  recommendation  to  allow  the  further 
development  of  a  materiel  solution  (Department  of  Defense  DoD,  2008).  The  decision 
maker  at  MDD  is  called  the  Milestone  Decision  Authority  (MDA).  The  MDA  reviews 
the  approved  ICD  and  any  proposed  concepts  to  ensure  that  the  material  solution 
decision  has  a  solid  foundation,  is  based  on  justified  information,  and  can  be 
developed  within  time  and  resource  constraints.  If  the  concepts  are  deemed 
adequately  mature,  they  proceed  to  the  next  phase  of  development  where  they  are 
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further  explored.  In  order  to  ensure  that  the  approved  concepts  are  adequately  mature, 
and  in  an  effort  to  encourage  more  rigor  at  this  early  gate,  the  MDA  can  draw  upon 
important  pieces  of  information  defined  herein  as  key  concept  maturity  elements. 

The  information  needed  by  the  MDA  at  Concept  Screening  is  largely 
associated  with  development  of  new  or  modified  systems  included  in  the  proposed 
concepts  (Figure  2).  These  concepts  involve  legacy  systems  and  any  anticipated 
changes  in  operations  and/or  materiel  to  existing  systems  should  be  identified.  A 
critical  piece  of  information  for  the  decision  maker  at  this  stage  is  the  scope  of  the 
required  changes  to  implement  the  concept,  since  later  decision  gates  will  be 
increasingly  associated  with  development  of  the  individual  component  systems  of  the 
concept  (CJCSM  3170.01D,  2009).  If  the  full  scope  of  the  concept  is  not  fully 
understood  prior  to  a  system  development  decision,  either  the  full  operational 
capability  will  not  be  realized,  or  significant  cost  impacts  will  be  forthcoming  to 
address  needed  modifications  to  other  systems. 

At  this  stage  of  concept  development  the  actual  proposed  solutions  are 
explained  in  terms  of  the  required  functionality  (SV-4,  SvcV-4),  and  the  relationship 
between  the  need  and  solution  is  defined  for  the  associated  systems  (SV-5,  SvcV-5). 
These  architecture  elements  may  have  been  initially  defined  for  existing  systems 
during  the  needs  identification  process  associated  with  Gate  1,  but  they  will  need  to  be 
updated  and  augmented  for  the  envisioned  modifications  and/or  developmental 
systems  associated  with  a  proposed  concept.  The  interfaces  and  relationships  between 
systems  and  services  should  also  be  updated  (SV-1-3,  SvcV-1-3).  The  technologies 
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critical  to  the  solution  need  to  be  identified  (SV-9)  and  any  known  standards 
applicable  to  the  concept  (StdV-1)  should  be  captured  (Hughes,  2010). 

The  level  of  detail  required  for  any  architecture  element  at  this  point  should  be 
driven  by  the  decision  at  hand  and  the  decision  maker  (Hughes,  2010).  At  the  concept 
screening  gate,  scope  and  problem  definition  dominate  the  decision  objectives,  and  the 
architecture  definition  to  support  this  decision  will  likely  require  no  more  than 
subsystem  identification  for  the  component  systems  of  the  concept.  Indeed,  novel 
solutions  considered  at  the  concept  screening  gate  will  not  likely  support  definition 
below  this  level.  Even  existing  solutions  where  detailed  information  is  available  will 
not  require  all  this  detail  be  included  in  the  architecture  products  at  this  early  stage. 
The  goals  of  this  phase  are  to  conduct  the  analyses  to  show  the  proposed  solution  will 
meet  the  identified  needs  and  to  identify  and  characterize  the  risks  associated  with  the 
solution. 

Concepts  that  are  allowed  to  proceed  through  the  screening  gate  will  need  to 
undergo  further  definition  and  will  eventually  have  to  compete  against  each  other. 

The  competition  is  in  the  form  of  a  cost/benefit  analysis  and  the  criteria  against  which 
the  concepts  are  measured  should  be  identified  prior  to  the  Concept  Screening  gate. 
Quantitative  target  specifications  (Measures  of  Performance)  that  describe  system 
characteristics  should  be  developed  prior  to  the  gate  for  use  in  the  analysis.  The  DoD 
conducts  an  Analysis  of  Alternatives  (AoA)  during  the  concept  refinement  phase  that 
acts  as  the  cost/benefit  analysis.  The  AoA  study  plan  sets  the  parameters  for  the 
critical  technologies  and  cost  drivers,  and  decision  objectives  for  the  AoA.  Sufficient 
risk  identification  associated  with  the  technologies,  interfaces,  or  changes  to  existing 
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systems  should  be  completed  to  ensure  that  the  AoA  further  addresses  all  pertinent 
issues  (Hughes,  2010). 

According  to  the  AoA  Handbook  (Office  of  Aerospace  Studies,  2008)  the  AoA 
study  plan  should  describe  how  the  following  questions  will  be  answered  in  the 
subsequent  Materiel  Solutions  Analysis  phase: 

•  What  alternatives  provide  validated  capabilities? 

•  Are  the  alternatives  operationally  effective  and  suitable? 

•  Can  the  alternatives  be  supported? 

•  What  are  the  risks  (technical,  operational,  programmatic)  for  each  alternative? 

•  What  are  the  life-cycle  costs  for  each  alternative? 

•  How  do  the  alternatives  compare  to  one  another? 

Due  to  the  uncertainty  associated  with  any  new  development,  risks  will  still  be 
present  in  a  program  or  project  regardless  of  the  level  of  risk  management.  However, 
if  an  ample  amount  of  rigor  is  applied,  the  more  expensive  risks  can  be  identified  and 
their  impacts  minimized.  Further  the  assertion  can  be  made  that  if  risks  are  identified 
and  sound  risk  mitigation  plans  are  developed,  a  much  more  realistic  idea  of  the 
resources  required  to  complete  development  will  be  produced,  thereby  resulting  in  a 
more  realistic  cost  estimate.  It  is  during  the  risk  identification  phase  when  the 
important  relationships  between  systems  engineering  and  system  architecture  are 
defined.  Risk  management  is  the  process  used  to  manage  the  uncertainty  associated 
with  new  development  as  described  by  Hillson  (2004);  therefore,  risk  management  is  a 
critical  element  of  concept  maturity. 

Information  Maturity  Elements  for  Gate  3. 

The  purpose  of  the  concept  selection  gate  is  to  evaluate  proposed  concepts  with 
respect  to  the  customer’s  needs  and  to  the  resources  required  to  develop  the  concept 
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(Ulrich  &  Eppinger,  2004).  Again,  this  is  similar  to  the  DoD  process  as  this  decision 
point  corresponds  with  Milestone  A  in  the  Defense  Acquisition  Framework. 

Following  a  successful  Milestone  A,  entry  into  a  technology  development  phase  is 
approved  for  one  or  more  prime  contractors.  The  purpose  of  this  phase  is  to  reduce 
technology  risk  and  to  determine  the  appropriate  set  of  technologies  to  be  integrated 
into  the  full  system  (Department  of  Defense,  2008).  The  program  formulation 
elements  associated  with  the  Milestone  A  are  defined  in  DoD  5000  series  and  the 
Defense  Acquisition  Guidebook  (Department  of  Defense),  but  additional  concept 
maturity  elements  will  be  discussed  here. 

In  both  industry  and  the  DoD,  an  investment  decision  must  be  made  based 
upon  the  information  and  analysis  contained  in  the  materiel  concept.  Much  of  this 
information  is  efficiently  and  effectively  conveyed  and  managed  via  architecture 
products.  Although  most  architecture  products  are  not  required  by  DoD  policy  until 
the  later  Milestone  B  decision  to  enter  a  detailed  design  phase,  the  NRC  study 
highlighted  several  important  benefits  to  earlier  development  of  systems  architecture: 

1 .  Architecture  can  mitigate  internal  and  external  system  complexity  risk  by 
partitioning  the  system  into  separately  definable  and  procurable  parts. 

2.  Architecture  can  reduce  lifecycle  costs  through  the  process  of  breaking 
down  large  systems  into  more  easily  managed  components  whereby  potential 
cost  and  schedule  risks  can  be  identified. 

3.  The  construction  of  a  rigorous  systems  architecture  developed  early  in  the 
program  will  aid  in  reducing  interface  complexity  control  problems  later  in  the 
program  when  they  are  much  more  costly  to  fix  (NRC,  2008). 

Some  architecture  elements  created  in  earlier  phases  will  need  to  be  updated 

and  others  will  require  dramatic  additions  in  the  phase  preceding  the  concept  selection 

decision  (Figure  2).  The  specifications  (MOP’s)  to  which  the  solution  will  be 
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measured  may  require  updating.  An  initial  identification  of  characteristics  or 
attributes  that  are  essential  for  the  development  of  the  capability  should  be  conducted. 
Mitigation  and  management  plans  will  need  to  be  created  for  previously  identified 
risks.  A  system  level  plan  to  develop  new  technologies  or  to  integrate  modified 
technologies  should  also  be  detailed  for  concept  selection.  The  DoD  calls  it  a  Systems 
Engineering  Plan  (SEP)  and  uses  it  to  describe  the  process  of  technology  maturation 
(ODUSD(A&T)SSE/ED,  2008).  The  SEP  provides  traceability  back  to  the  users’ 
needs  via  elements  such  as  a  CONOPs,  risk  identification  and  architecture  definition 
eventually  focusing  attention  towards  a  preferred  system  concept.  The  final  version  of 
the  overall  concept  should  be  sufficient  to  characterize  how  the  solution  will  meet  the 
identified  need,  to  characterize  the  amount  of  risk  involved  with  developing  the 
solution,  and  to  give  a  reasonable  idea  of  the  full  cost  associated  with  the  proposed 
solution.  This  work  describes  how  the  right  combination  of  architecture  views  along 
with  the  other  aforementioned  maturity  elements,  can  mature  a  concept  relative  to  the 
early  decision  points  in  the  development  process. 

The  following  chapter  will  discuss  a  proposed  methodology  for  validation  of 
the  concept  maturity  framwork  and  its  Concept  Evaluation  and  Selection  within  a 
Stage-Gated  Process.  The  process  focuses  on  proving  a  time -phased  element  driven 
framework  and  its  ability  to  correctly  diagnose  a  healthy  level  of  concept  maturity  at 
the  needed  time  and  phase.  This  validation  effort  will  be  based  on  the  maturity 
elements  discussed  in  this  chapter  and  as  presented  in  Hughes’  Framework. 
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III.  Methodology 


A.  Research  Approach 

The  approach  and  methodology  used  by  the  researchers  is  focused  on 
addressing  the  research  questions  and  the  main  hypothesis  that  the  Hughes’ 

Framework  adds  value  to  understanding  a  concept’s  maturity  in  early  development. 
Along  with  the  validation  effort,  the  research  team  was  looking  for  areas  to  improve 
upon  the  framework  if  deemed  appropriate.  The  purpose  of  this  research  is  both 
summative  and  formative  in  nature.  The  first  part  of  this  methodology  is  summative  in 
that  the  researchers  were  summing  up  conclusions  about  the  framework  in  order  to 
recommend  and  comment  on  its  value  (Patton,  2002;  Krathwohl,  1998).  The  second 
part  is  formative,  aimed  at  improving  the  framework  if  indicated  by  the  data  analysis 
and  results  (Patton,  2002;  Krathwohl,  1998).  In  an  effort  to  increase  the  strength  of 
both  the  evidence  and  generated  findings,  the  researchers  chose  the  principle  of 
triangulation  as  a  method  to  improve  the  study’s  rigor  and  believability. 

1.  Triangulation. 

Triangulation  strengthens  a  study  by  combining  different  methods,  data  types, 
theories,  and  perspectives  during  the  research  effort  (Patton,  2002;  Denzin,  1978). 

This  triangulation  approach  prompted  a  framework  validation  effort  that  would 
include  (1)  interviews  with  knowledgeable  individuals  experienced  in  early  concept 
development,  (2)  an  attempt  to  apply  the  framework  to  a  real-world  concept  in 
development,  and  (3)  an  analysis  of  approved  policy  and  guidance  on  items  related  to 
the  maturity  elements  of  the  framework.  This  methodology  will  help  increase  the 
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confidence  that  any  findings  or  conclusions  made  are  supported  by  various  angles  and 
reduce  the  potential  bias  or  errors  from  a  single  approach  (Patton,  2002). 

2.  Internal  &  External  Validity. 

As  a  way  to  check  the  believability,  level  of  significance,  and  ability  to 
generalize  the  finding  of  this  research,  the  concepts  of  internal  and  external  validity  are 
leveraged  (Krathwohl,  1998).  Internal  validity  is  the  power  of  a  study  to  support  a 
cause  linking  to  an  effect  (Krathwohl,  1998).  Interviews  will  be  the  mechanism  used 
to  test  internal  validity.  The  cause  will  be  the  interviewee’s  responses  regarding  the 
framework  linking  to  the  effect  of  a  conclusion  that  supports  or  negates  usage  of  the 
framework.  The  researchers  will  apply  the  five  judgments  as  prescribed  by 
Krathwohl  (1998)  to  test  for  internal  validity.  Please  see  Table  3  below.  If  the 
interview  method  results  in  a  favorable  recommendation  for  the  framework  and  the 
method  is  determined  to  be  internally  valid,  then  the  researchers  will  test  for  external 
validity. 

External  validity  is  the  power  of  a  study’s  findings  to  be  generalized  to  other 
areas  outside  of  the  controlled  study  (Krathwohl,  1998).  In  terms  of  this  research,  a 
test  for  strong  external  validity  will  show  how  the  framework  not  only  adds  value  to 
the  subjects  interviewed,  but  would  add  value  to  other  organizations  as  well. 

Although,  the  interviews  focused  on  subjects  with  backgrounds  primarily  in  Air  Force 
product  development,  the  researchers  will  demonstrate  the  external  validity  of  this 
study’s  findings  as  they  apply  to  the  greater  DoD  and  even  commercial  product 
development  organizations.  Both  the  application  and  confirmatory  sources  approach 
of  this  methodology  will  be  used  to  help  test  for  the  degree  to  which  the  framework 
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can  be  generalized.  Similar  to  internal  validity,  Krathwohl  (1998)  prescribes  five 
separate  judgments  that  the  research  team  will  use  to  test  for  the  external  validity  of 
this  study’s  findings,  see  Table  3  below. 


Table  3  -  Five  Judgments  for  Internal  &  External  Validity 


Internal  Validity 

External  Validity 

1 .  Explanation  credibility 

1 .  Explanation  generality 

2.  Translation  fidelity 

2.  Translation  generality 

3.  Demonstrated  result 

3.  “Demonstrated  generality” 

4.  Rival  explanations  eliminated 

4.  Restrictive  explanations  eliminated 

5.  Credible  result 

5.  Replicable  result 

(Krathwohl,  1998) 

B.  Triangulation  Approach:  Interview  and  Associated  Methodology 

1.  Objectives  &  Reasoning. 

The  Hughes’  Framework  is  a  recent  effort  to  assess  concept  maturity  during 
early  development.  As  part  of  this  validation  effort,  the  research  team  determined  that 
individuals  actively  or  recently  involved  in  related  early  development  would  best  be 
suited  to  assess  the  framework’s  value.  These  identified  individuals  are/were  most 
engaged  in  the  triumphs  and  pitfalls  of  the  work  involved  with  early  development  and 
capturing  their  perceptions  of  what  is  important  to  concept  maturity  is  essential. 
Although  the  Hughes’  Framework  is  bom  out  of  scholarly  research,  there  is  no  proxy 
or  substitute  to  compare  it  directly  to  in  order  to  assess  its  value  or  correctness.  For 
this  reason,  the  research  team  avoided  using  a  comparison  to  accepted  frameworks  or 
related  guidance  on  concept  maturity  as  the  primary  method  of  validation.  The 
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interview  methodology,  therefore,  will  serve  as  the  main  cause  for  recommending 
acceptance  or  refusal  of  the  framework.  Likewise,  the  research  team  has  the  objective 
to  ensure  the  methodology  and  findings  of  the  interview  approach  are  both  internally 
and  externally  valid. 

2.  Design. 

The  research  team  framed  the  interview  to  gather  both  quantitative  and 
qualitative  data  as  a  method  for  analysis  and  validation.  The  quantitative  data  would 
show  the  ratio  of  respondents  voting  favorably  for  or  against  the  maturity  elements 
within  the  framework  while  the  qualitative  data  would  allow  for  further  analysis  and 
uncover  the  rationale  and  context  behind  the  interviewee’s  responses  (Patton,  2002). 
The  quantitative  data  would  also  help  facilitate  direct  comparisons  and  contrasts 
between  respondents  (Patton,  2002).  Krathwohl  (1998)  offers  that  quantitative 
methods  alone  are  inadequate  when,  “detailed,  in-depth  information  is  sought .  .  . 

[and]  you  believe  the  perceptions  of  the  participants  differ  from  those  of  outside 
observers”  (Krathwohl,  1998,  p.243).  This  level  of  detail  and  perception  of  the 
interviewees  is  critical  to  fully  understand  what  is  important  during  early  development, 
a.  Subjects. 

Given  the  need  for  both  quantitative  and  qualitative  data  concerning  the 
framework,  the  research  team  next  determined  the  appropriate  subjects  to  interview. 
The  researchers  approached  choosing  subjects  from  several  angles,  but  it  was 
concluded  that  purposeful  homogenous  sampling  would  be  the  best  option. 

Purposeful  sampling  has  the  benefits  of  focusing  one’s  research  effort  on  information- 
rich  cases  (Patton,  2002;  Krathwohl,  1998),  while  homogenous  sampling  helps  reduce 
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variation  and  the  noise  often  associated  with  random  sampling  (Patton,  2002).  The 
research  team  wanted  to  restrict  the  opinions  on  the  framework’s  value  and  usefulness 
to  individuals  actively/recently  involved  in  early  development  and  avoid  individuals 
that  are  not  specialized  in  this  field.  This  purposeful  sampling  may  appear  too  biased 
by  selecting  such  a  homogenous  subject  base,  but  Patton  claims,  “what  would  be  ‘bias’ 
in  statistical  sampling,  and  therefore  a  weakness,  becomes  intended  focus  in 
qualitative  sampling,  and  therefore  a  strength”  (Patton,  2002,  p.230). 

In  an  effort  to  identify  the  individuals  desired  for  sampling,  the  research  team 
created  the  concept  of  a  practitioner.  A  practitioner  would  be  the  ideal  candidate  to 
interview.  For  the  purposes  of  this  research  a  practitioner  can  be  defined  as  an 
individual  who  actively  or  recently  prepares,  constructs,  designs,  or  manages  the 
information  and  activities  involved  in  early  concept  development.  These  individuals 
are  the  most  experienced  and  knowledgeable  people  that  understand  how  early 
development  is  done  currently  or  in  the  recent  past,  and  can  recognize  what 
information  decision-makers  need  or  desire  to  assess  a  concept’s  maturity.  The 
insights  of  decision-makers  would  also  be  beneficial,  but  the  research  team  recognized 
the  extremely  small  sample  of  high-level  decision-makers  available  in  the  DoD  and 
the  difficulty  accessing  them. 

b.  Sample  Size. 

The  sample  size  was  not  a  major  concern  during  this  study  as  mentioned 
previously  regarding  the  subject  base;  the  research  team  wanted  the  right  people  to 
focus  the  research  on  instead  of  the  quantity.  For  this  reason  and  other  constraints 
such  as  the  restrictive  time  to  finish  the  study,  limited  travel  resources,  and  willingness 
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of  participants,  the  sample  size  would  be  lower  than  other  social  science  research. 
However,  Patton  (2002)  offers  that  unlike  statistical  sampling,  research  that  is  highly 
qualitative,  .  .  [has]  more  to  do  with  the  information  richness  of  the  cases  [subjects] 
selected  and  the  observational/analytical  capabilities  of  the  researcher  than  with  the 
sample  size”  (p.245).  Accordingly,  the  research  team  was  willing  to  limit  the  sample 
size  to  focus  on  information-rich  candidates.  Nevertheless,  in  an  effort  to  limit  the 
bias  of  any  single  organization  within  the  Air  Force,  the  research  team  sought  to 
broaden  the  subject  base  and  avoid  interviewing  a  largely  disproportionate  number  of 
individuals  from  the  same  organization.  The  research  team  recognizes  the  small 
sample  size  as  a  relevant  target  for  criticism  and  a  limitation  of  the  research. 
Regardless,  in  keeping  with  Patton,  the  data  and  results  from  a  small  sample  base  can 
yield  substantial  findings  with  the  appropriate  care  and  rigor.  These  concerns  will  be 
at  the  forefront  of  this  study’s  data  analysis. 

c.  Structured  Interview. 

The  interview  was  framed  to  gather  the  perceptions  and  viewpoints  of  the 
value  of  the  maturity  elements  within  the  framework  as  well  as  any  relevant  discussion 
on  related  topics  such  as  early  development  and  acquisitions.  The  intent  was  to  gather 
both  quantitative  data  for  consistency  in  results  between  respondents  and  qualitative 
data  to  promote  a  deeper  understanding.  With  these  goals  in  mind,  the  researchers 
determined  a  structured  interview  approach  as  recommended  by  Krathwohl  (1998) 
would  be  best.  This  approach  would  include  asking  the  interviewee  common 
questions  regarding  the  framework  while  allowing  them  time  for  follow-on  discussion. 
A  dilemma  arose  in  that  the  research  team  did  not  want  to  artificially  skew  the 
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responses  by  pre-supposing  a  notion  that  the  Hughes’  Framework  is  useful  by  only 
asking  questions  about  the  framework  itself.  The  framework  as  depicted  in  Figure  2 
(Page- 19),  requires  a  level  of  explanation  and  background  to  fully  understand  its  usage 
or  value.  Simply  showing  the  framework  to  respondents  and  asking  for  their  thoughts 
would  require  some  explanation.  Furthermore,  as  this  study’s  authors  are  not  the 
authors  of  the  Hughes’  Framework,  a  risk  would  lie  in  the  ability  of  the  team  to 
articulate  properly  how  the  framework  operates.  This  weakness  could  possibly  result 
in  an  interviewee’s  misunderstanding  of  the  Hughes’  framework.  Therefore,  the  focus 
was  placed  on  whether  or  not  a  respondent  perceived  value  in  the  content  within  the 
framework  rather  than  merely  its  design  and  appearance.  Finding  a  method  to 
properly  gain  the  best  possible  unbiased  information  from  each  respondent  with  a 
mixture  of  both  qualitative  and  quantitative  data  became  the  priority  and  was 
addressed  through  studying  and  applying  the  methodology  linked  to  both  preference 
measurement  and  conjoint  analysis. 

d.  Preference  Measurement  &  Conjoint  Analysis. 

Preference  measurement  as  discussed  by  Netzer  et  al.  (2008)  is  the  concept 
related  to  assessing  a  subject’s  likes,  dislikes,  and  general  degree  of  perceived  value. 

A  closely  related  concept,  conjoint  analysis,  is  a  method  commonly  applied  in  market 
research  when  marketers  want  to  determine  what  features  of  a  product  a  customer 
wants  or  would  purchase  (Dahan  &  Hauser,  2001a).  Netzer  et  al.  (2008)  comments 
that  in  the  past,  preference  measurement  was  nearly  synonymous  with  conjoint 
analysis  in  determining  what  features  consumers  preferred  in  a  product.  In  their  recent 
studies,  Netzer  et  al.  and  Michalek,  Feinberg,  and  Papalambros  (2005)  claim  that 
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advances  in  preference  measurement  methods  have  evolved  to  include  more  than  just 
marketing  for  companies.  As  mentioned  by  Netzer  et  al.  the  utility  and  breadth  of 
methods  for  both  preference  measurement  and  conjoint  analysis  has  expanded 
significantly.  The  methods  and  benefits  are  now  far-reaching  as  further  research  has 
extended  into  policy-making,  health  care,  engineering  and  academic  research  (Netzer 
et.  al,  2008;  Micahelek  et.  al,  2005) 

The  research  team  used  principles  of  preference  measurement  and  conjoint 
analysis  to  develop  the  interview  approach  used  to  discover  the  preferences  of  the 
respondents  concerning  their  perceived  value  of  the  maturity  elements  within  the 
framework.  In  a  study  regarding  the  utility  of  conjoint  analysis  towards  customers 
selecting  preferences  for  vehicle  features  in  a  web-based  format,  Dahan  and  Hauser 
(2001b)  use  cards  with  different  representations  of  vehicles  and  ask  the  test  subjects  to 
select  the  cards  they  would  most  likely  purchase  by  rank  ordering  them.  In  an  effort  to 
not  only  gain  insight  into  what  maturity  elements  the  respondents  for  this  validation 
effort  of  the  framework  value,  the  research  team  also  desired  to  understand  any 
potential  value  ranking  or  ordering  of  the  elements.  Consequently,  the  team  chose  to 
extend  the  method  used  in  the  Dahan  and  Hauser  (2001b)  study, 
e.  Card  Selection  Exercises. 

The  research  team  chose  to  use  the  idea  of  the  respondents  selecting  cards  that 
represented  maturity  elements  in  the  Hughes’  Framework  during  different  exercises 
within  the  interview.  These  exercises  were  meant  to  harness  the  concepts  inherent  to 
conjoint  analysis,  and  give  an  avenue  for  discussion  and  interaction  with  the 
respondents.  As  mentioned  previously,  the  research  team  wanted  to  focus  the 
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interview  on  the  content  of  the  framework  rather  than  over  its  appearance  or  style  and 
avoid  relying  on  the  ability  of  the  researchers  to  properly  explain  the  framework  and 
possibly  biasing  the  subjects.  While  discussing  conjoint  analysis  in  his  article,  Marder 
(1999)  claims  that  any  conjoint  analysis  study  has  a  major  assumption,  that  the  overall, 
.  .  value  of  a  product  is  the  aggregation  of  the  values  of  its  characteristics”  (p.2-3). 
Marder’s  findings  support  the  approach  that  validating  the  value  of  the  framework  (the 
product)  will  be  determined  by  aggregating  the  value  of  the  individual  maturity 
elements  (its  characteristics). 

After  determining  that  breaking  up  the  framework  into  its  maturity  elements 
would  be  a  logical  approach  towards  validation,  the  research  team  reviewed  different 
types  of  conjoint  analysis  methods  discussed  within  a  study  by  Hauser  and  Rao  (2003) 
in  order  to  find  the  best  technique  to  conduct  the  interviews.  The  authors  offer  that 
more  and  more  methods  and  techniques  are  evolving  with  time  as  well  the  notion  of 
hybrid  methods,  which  combine  and  tweak  some  traditional  methods.  Hauser  and  Rao 
further  claim  that  no  method  is  better  than  another  as  they  all  have  strengths  and 
weaknesses,  and  that  tailoring  the  method  to  fit  the  research  is  what  is  important.  In 
another  conjoint  analysis  application  study,  Dahan,  Hauser,  Simester,  and  Toubia 
(2002)  claim  that  empirical  evidence  exists  that  hybrid  combinations  of  conjoint 
analysis  methods,  “often  yield  more  accurate  or  more  efficient  predictions  than  either 
of  the  parent  methods”  (p.20).  Dahan  and  Hauser  (2001b)  also  claim  that  hybrid 
methods  work  well  with  research  that  is  intended  to  measure  the  intensity  of  a 
preference. 
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The  conjoint  analysis  methods  selected  for  this  framework  validation  effort  are 
a  hybrid  mix  of  a  full  profile  evaluation  and  the  self-explicated  method.  In  the  full 
profile  evaluation  method,  the  features  describe  the  product  and  the  respondent  is 
usually  asked  to  rank  order  or  give  a  preference  rating  for  all  the  product  variations 
based  on  their  features  (Hauser  &  Rao,  2003).  Using  the  vehicle  example,  this  method 
could  have  respondent’s  preference  rank  different  configurations  of  a  vehicle  such  as  a 
mid-sized  sedan  with  four-wheel  drive,  six-cylinder  engine,  and  leather  interior  to  a 
two-wheel  drive,  six-cylinder  with  cloth  interior.  In  contrast,  the  self-explicated 
method  asks  respondents  more  about  the  features  themselves  than  the  products  they 
describe  in  an  effort  to  understand  the  relative  value  behind  the  features  (Hauser  & 
Rao,  2003).  Using  the  vehicle  example  this  method  could  have  respondents  describe 
what  they  like  most  about  certain  features  such  as  why  they  prefer  leather  to  a  cloth 
interior  or  why  they  like  leather  in  general.  In  relation  to  this  validation  effort  a  hybrid 
of  these  methods  became  the  basis  for  the  dual  cards  exercise  approach  to  the 
interviews.  Rather  than  features  of  a  product  to  purchase  as  used  in  the  Dahan  and 
Hauser  (2001b)  study  with  vehicle  preferences,  the  research  team  placed  the 
information  regarding  maturity  elements  within  the  framework  on  the  cards. 

The  interview  asks  each  respondent  for  some  background  information  to  obtain 
context  for  the  types  and  experiences  of  the  individuals  surveyed.  Some  of  the 
specifics  to  this  background  information  are  masked  to  protect  the  anonymity  of  the 
respondents.  After  the  background  information,  the  researchers  followed  with  the  first 
of  two  exercises.  During  and  after  each  exercise,  the  researchers  asked  related  follow- 
on  questions.  Due  to  the  short  duration  allotted  for  the  interviews,  the  research  team 
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chose  to  limit  the  number  and  scope  of  all  follow-on  questions  and  attempted  to 
refocus  any  discussion  that  was  overly  detailed.  Furthermore,  the  researchers  did  not 
want  any  respondent  focusing  too  much  attention  on  any  one  area  at  the  risk  of  losing 
all  detail  in  others.  Although  much  more  data  and  information  could  have  been 
gathered  in  these  interviews,  during  their  conjoint  analysis  study,  Dahan  and  Hauser 
(2001b)  claim,  “.  .  .  due  to  respondent  wear  out,  accuracy  degrades  as  the  number  of 
questions  increases  (p.340).  For  these  reasons,  the  researchers  chose  a  streamlined  and 
focused  approach  to  constructing  and  applying  the  interview  and  exercises.  Please 
reference  Appendix  A,  for  the  structure  and  questions  asked  during  the  interviews. 

The  first  exercise,  the  “As-Is”  process,  focused  on  the  principles  behind  the 
self-explicated  method  and  sought  to  understand  how  each  respondent  perceives  to 
what  extent  the  maturity  elements  are  being  accomplished  now  or  recently  during  the 
early  stages  of  development  (i.e.  post-  need  identification  and  pre-  MS-A).  In 
response  to  Drazen’s  (2004)  opinions  on  subjective  data,  each  respondent  was  asked  to 
not  think  about  what  the  correct  answer  is,  should  be  or  what  other  people  answered, 
and  that  the  focus  of  the  study  for  all  questions  is  completely  based  on  their 
perceptions.  In  order  to  keep  the  context  of  the  data  as  consistent  as  possible,  each 
respondent  was  asked  to  focus  their  perceptions  based  on  a  time  range  from  five  years 
prior  to  the  present.  Since  the  self-explicated  method  focuses  on  the  features  as 
mentioned  by  Hauser  and  Rao  (2003),  the  researchers  chose  to  ask  each  respondent  to 
describe  how  well,  how  bad,  how  often,  how  difficult,  or  any  related  comments 
concerning  the  information  presented  on  the  cards.  This  approach  would  lead  to  an 
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open,  yet  focused  discussion  to  gather  qualitative  data  on  the  perceptions  of  the 
maturity  elements  as  they  are  accomplished  now  or  recently. 

The  second  exercise,  the  “To-Be”  process,  focused  on  using  the  full  profde 
evaluation  method,  discussed  in  2003  by  Hauser  and  Rao,  to  ask  each  respondent  to 
pick  the  maturity  elements  they  deemed  as  adding  value  during  early  development. 
The  respondents  were  asked  to  select  any  cards  they  perceived  as  adding  value  during 
early  development  and  provide  qualitative  comments  and  any  rationale.  The 
respondents  were  told  that  the  second  exercise  is  independent  of  the  first  exercise  and 
that  they  should  only  select  cards  they  perceive  as  adding  value  regardless  of  if  it  is 
presently  being  accomplished.  The  respondents  were  also  asked  to  describe  an 
ordering  or  grouping  for  the  cards  they  think  are  most  efficient  and  effective  at  adding 
value  as  information  to  a  decision  maker.  To  limit  any  bias,  the  research  team 
reminded  each  respondent  that  the  researchers  neither  had  any  pre-conceived  opinion 
as  to  the  value  of  the  cards  nor  that  an  ordering  or  grouping  is  necessary.  Similar  to 
the  first  exercise,  the  researchers  asked  focused  follow-on  questions  such  as  what  are 
the  three  most  important  activities  for  this  new  set  of  cards  and  if  there  is  anything  the 
respondent  feels  is  missing  or  any  additional  information  they  would  include.  Finally, 
the  interview  concluded  with  time  for  each  respondent  to  provide  any  additional 
comments  concerning  the  cards,  early  product  development,  acquisitions  in  general, 
and/or  any  related  comments. 

f.  Mapping  to  Maturity  Elements. 

Before  the  interviews  and  data  collection  could  begin,  the  researchers  needed  a 
systematic  process  to  break  apart  the  Hughes’  Framework  into  its  maturity  elements 
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that  could  then  be  presented  to  the  respondents  for  discussion.  An  issue  arose  with 
simply  presenting  each  maturity  element  directly  from  the  framework  due  to  the 
brevity  of  descriptions  as  well  as  the  high  number  of  elements.  The  Hughes’ 
Framework  as  discussed  previously  leverages  the  use  of  the  DoDAF  architecture 
products  to  describe  many  of  the  maturity  elements.  The  researchers  did  not  want  to 
bias  the  respondents  by  asking  them  to  assess  the  value  or  current  use  of  the  DoDAF 
products  directly.  Furthermore,  the  descriptions  for  some  of  DoDAF  products  are 
lengthy  and  complex  which  could  cause  unnecessary  confusion  and  discussion  on  the 
purpose  behind  each  maturity  element.  For  these  reasons,  the  researchers  chose  to 
mask  the  use  of  any  DoDAF  products  within  the  exercise  cards  and  attempted  to 
simplify  the  maturity  elements  as  much  as  possible. 

Besides  the  desire  to  simplify  the  information  presented  on  the  cards  to  avoid 
confusion,  the  researches  attempted  to  reduce  the  overall  number  of  maturity  element 
cards  as  recommended  by  the  Dahan  and  Hauser  (2001b)  study.  Their  study  focused 
on  the  customers  selecting  their  preferred  vehicles  represented  by  cards  with 
information  on  them  using  a  web  interface,  and  they  determined  that  customers  had  a 
difficult  time  rating  vehicles  as  the  number  of  cards  increased  (Dahan  &  Hauser, 
2001b).  By  consolidating  similar  and  related  maturity  elements  within  the  framework, 
the  researchers  were  able  to  reduce  the  number  of  cards  used  from  24  to  14,  while  still 
maintaining  the  original  intent  and  necessary  information.  Two  additional  cards, 
which  represent  information  required  after  the  scope  of  the  Hughes’  Framework,  were 
added  by  the  research  team  to  gain  insight  into  the  reasoning  for  their  exclusion.  The 
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mapping  of  the  maturity  elements  to  these  16  cards  used  in  the  exercise  and  the 
rationale  for  the  two  additional  cards  is  explained  below. 

The  method  for  the  mapping  was  to  group  similar  maturity  elements  from  the 
framework  under  a  more  general  and  easily  understood  description.  Further,  the 
grouping  often  involved  similar  maturity  elements  for  different  stages  described  in  the 
framework.  Since  the  purpose  of  the  interview  was  again  to  focus  on  the  content 
within  the  framework,  the  researchers  determined  that  generalizing  the  phases  would 
not  skew  the  data.  These  cards  were  labeled  for  identification  purposes  from  “A”  to 
“P”.  During  the  explanation  of  the  interview  process,  the  respondent  was  instructed  to 
ignore  the  labels  and  that  it  has  no  significance  other  than  for  recording  purposes. 
Finally,  two  cards,  “J”  and  “O”,  represent  common  activities  and  products  performed 
during  development  and  acquisitions,  but  were  not  included  in  the  framework.  The 
researchers  chose  to  add  these  cards  to  hopefully  gain  some  insight  into  the  reasoning 
for  their  exclusion  from  the  Hughes’  framework  and  allow  for  further  discussion  from 
the  audience.  The  actual  cards  used  during  the  interviews  are  depicted  in  Appendix  B. 
The  following  section  details  and  describes  the  mapping  process  for  each  card  and 
Table  4,  below  summarizes  the  mapping. 

A.  Rough  order  magnitude  initial  cost  and  schedule  estimates.  This  is  not  a 
DoDAF  architectural  document  specifically  called  out  in  the  framework  but 
a  combination  of  many  elements  in  the  framework  that  are  the  results  of  the 
analysis  that  is  performed.  In  the  concept  screening  phase  this  card 
represents  the  “Cost/Benefit  Analysis  Guidance”  and  “Initial  Estimation  of 
resources  required  for  new  development  and  modification  of  existing 
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systems”  maturity  elements.  The  card  also  represents  the  “Cost/Benefit 
Analysis  Results”  and  “Estimations  of  resources  required  for  development, 
operation  and  sustainment  associated  with  the  proposed  solution”  during 
the  concept  selection  phase. 

B.  Description  of  interfaces  between  system  elements  and  the  resources  that 
flow  between  them,  as  well  as  the  operational  activities  they  support.  This 
card  represents  OV-2,  SV-2  and  SV-3  DoDAF  architecture  products.  In 
the  framework  this  would  be  the  “Systems  Interfaces”  and  “Operational 
Activity”  maturity  elements  found  in  the  concept  screening  phase. 

C.  List  of  assumptions,  constraints  and  enabling  capabilities  that  are  required 
for  the  system  to  operate  effectively.  This  card  combines  sections  of  a 
CONOPS  and  an  OV-6a  DoDAF  architectural  product.  The  CONOPS 
helps  describe  the  assumptions,  constraints,  and  enabling  capabilities; 
while  an  OV-6a  helps  supplement  with  additional  business  rules  and 
constraints  the  system  must  operate  under.  In  the  framework  this  would  be 
the  “Operational  Constraints”  maturity  element  in  the  opportunity 
identification  phase. 

D.  Organizational  relationships,  their  roles  and  responsibilities,  and  how  the 
flow  of  resources  is  expected  to  occur.  This  card  represents  an  OV-4 
DoDAF  architectural  product.  In  the  framework  this  would  be  the 
“Organizations  &  Services”  maturity  element  in  the  opportunity 
identification  phase. 
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E.  A  decomposition  of  all  operational  activities  or  tasks,  and  the 
inputs/outputs  between  them.  This  card  represents  0V-5a  and  0V-5b 
DoDAF  architectural  products.  In  the  framework  this  would  be  the 
“Operational  Activity”  maturity  element  in  the  concept  screening  phase. 

F.  Identification  of  the  system  as  a  whole,  the  elements  that  make  up  the 
system  and  their  interconnections  internally  and  external  to  the  system. 

This  card  represents  an  SV-1  DoDAF  architectural  product.  In  the 
framework  this  would  be  the  “Existing  Systems-Interfaces”  maturity 
element  in  the  opportunity  identification  phase  as  well  as  the  “Impact  to 
other  systems”  and  “Systems  interface”  maturity  elements  in  the  concept 
screening  phase. 

G.  A  description  of  the  functions  performed  by  systems  and  the 
interactions/resources  flows  required  to  perform  that  function.  This  card 
represents  an  SV-4  DoDAF  architectural  product.  In  the  framework  this 
would  be  the  “Form  &  Function”  maturity  element  in  the  concept  screening 
phase. 

H.  A  mapping  of  desired  system  capabilities  to  operational  activities  that  must 
be  performed  and  the  system  functions  that  support  them.  This  card 
represents  the  SV-5  and  CV-6  DoDAF  architectural  products.  In  the 
framework  this  would  be  the  overlapping  of  the  “Operational  Activities 
Gap”  and  the  “Existing  Systems  Interfaces”  maturity  elements  in  the 
opportunity  identification  phase  as  well  as  the  interactions  between  the 
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“Systems  Interfaces,  Operational  Activity  and  Form  &  Function”  maturity 
elements  in  the  concept  screening  phase. 

I.  A  listing  of  maturing  technologies  that  the  development  of  the  system  may 
be  dependent  upon.  This  card  represents  a  SV-9  DoDAF  architectural 
product.  In  the  framework  this  would  be  the  “Technology”  maturity 
element  in  the  concept  screening  phase  and  the  “Technology  Development 
Plans”  in  the  concept  selection  phase. 

J.  A  listing  of  all  industry,  national  and  international  standards  that  apply  to 
the  system  (solution.)  This  card  represents  a  StdV-1  DoDAF  architectural 
product.  This  is  not  called  out  in  the  framework  but  since  it  is  a  DoDAF 
product  the  interviewee’s  reaction  on  whether  it  should  impact  the  decision 
maker’s  choice  could  provide  some  valuable  information  for  the  research. 

K.  A  listing  of  capabilities  needed  to  solve  the  problem  decomposed  down  to 
lower  level  enabling  capabilities  and  the  dependencies  between  them.  This 
card  represents  the  CV-2  and  CV-4  DoDAF  architectural  products.  In  the 
framework  this  would  be  represented  by  the  “Operational  Activity  Gap” 
maturity  element  in  the  opportunity  identification  phase  and  the 
“Operational  Activity”  maturity  element  in  the  concept  screening  phase. 

L.  A  matrix  matching  required  system  capabilities  to  the  operational  activities 
that  must  be  performed  to  achieve  them.  This  card  represents  the  SvcV- 1 
through  SvcV-5  DoDAF  architectural  products.  In  the  framework  this  is 
represented  by  the  “Organizations  &  Services”  maturity  elements  in  the 
opportunity  identification  and  concept  screening  phases. 
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M.  A  risk  assessment  matrix  of  the  risky  elements  of  system  development  and 
integration  as  well  as  a  mitigation  strategy.  This  is  not  a  DoDAF 
architectural  document  specifically  called  out  in  the  framework  but  a 
combination  of  many  elements  in  the  framework  that  are  the  results  of  the 
analysis  that  is  performed.  In  the  opportunity  identification  phase  this 
represents  the  “Operational  Risk”  maturity  element.  In  the  concept 
screening  this  represents  the  “Initial  identification  of  development  risk” 
maturity  element.  In  the  concept  selection  phase  this  represents  the  “Risk 
Management  Plans”  maturity  element. 

N.  Quantifiable  measures  of  effectiveness  (MOE)for  the  system  and  derived 
measures  of  performance  (MOP.)  This  card  represents  some  of  the 
information  that  is  contained  in  an  AV-1  DoDAF  architectural  product. 

This  represents  and  combines  parts  of  the  “CONOPS  Mission  /  Objectives  / 
Tasks  /  Measures  (MOE)”  maturity  element  in  the  opportunity 
identification  phase,  the  “Updated  Target  Specifications  (MOP)”  maturity 
element  in  the  concept  screening  phase  and  the  “Changes  to  Target 
Specifications  (MOP)”  and  “Initial  selection  of  Key  Performance 
Parameters  (KPP)”  maturity  elements  in  the  concept  selection  phase. 

O.  Initial  test  plan  for  how  to  evaluate  the  system  against  its  MOPs  and 
MOEs.  This  card  does  not  represent  any  maturity  element  in  the 
framework  nor  is  it  specifically  called  out  as  a  DoDAF  architecture 
product.  This  card  was  added  in  the  interview  to  gather  reactions  on 
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whether  or  not  a  test  plan  should  be  a  factor  for  a  decision  maker  early  on 


when  choosing  between  concepts. 

P.  Concept  of  operations  (CON OPS)  describing  the  problem,  the  desired 
effects,  assumptions,  critical  capabilities,  enabling  capabilities,  sequenced 
actions  and  the  end  state.  This  card  represents  more  than  just  an  OV-1 
DoDAF  architectural  product.  In  the  framework  this  represents  the 
“CONOPS  Mission  /  Objectives  /  Tasks  /  Measures  (MOE)”  maturity 
element  in  the  opportunity  identification  phase. 


Table  4  -  Mapping  of  Exercise  Cards  Summary 


Exercise 

Card 

Phase 

Maturity  Element  from  Hughes’ 
Framework 

Related 
Architecture  / 
Acquisitions 
Products 

A 

Concept 

Screening 

Cost/Benefit  Analysis  Guidance;  Initial 
Estimation  of  resources  required  for  new 
development  and  modification  of  existing 
systems 

Concept 

Selection 

Cost/Benefit  Analysis  Results; 

Estimations  of  resources  required  for 
development,  operation  and  sustainment 
associated  with  the  proposed  solution 

B 

Concept 

Screening 

Systems  Interfaces;  Operational  Activity 

OV-2,  SV-2,  SV-3 

C 

Opportunity 

Identification 

Operational  Constraints 

CONOPS,  OV-6a 

D 

Opportunity 

Identification 

Organizations  &  Services 

OV-4 

E 

Concept 

Screening 

Operational  Activity 

OV-5a,  OV-5b 

F 

Opportunity 

Identification 

Existing  Systems  Interfaces 

SV-1 

Concept 

Screening 

Systems  Interface 
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Exercise 

Card 

Phase 

Maturity  Element  from  Hughes’ 
Framework 

G 

Concept 

Screening 

Form  &  Function 

H 

Opportunity 

Identification 

Operational  Activities  Gap;  Existing 
Systems  Interfaces 

Concept 

Screening 

Systems  Interfaces;  Operational  Activity; 
Form  &  Function 

I 

Concept 

Screening 

Technology 

Concept 

Selection 

Technology  Development  Plans 

J 

*not  in  Hughes’  Framework 

K 

Opportunity 

Identification 

Operational  Activity  Gap 

Concept 

Screening 

Operational  Activity 

L 

Opportunity 

Identification 

Organizations  &  Services 

Concept 

Screening 

Organizations  &  Services 

Opportunity 

Identification 

Operational  Risk 

M 

Concept 

Screening 

Initial  Identification  of  Development 

Risk 

Concept 

Selection 

Risk  Management  Plans 

Opportunity 

Identification 

CONOPS  Mission  /  Objectives  /  Tasks  / 
Measures  (MOE) 

N 

Concept 

Screening 

Updated  Target  Specifications  (MOP) 

Concept 

Selection 

Changes  to  Target  Specifications  (MOP); 
Initial  Selection  of  Key  Performance 
Parameters  (KPP) 

O 

*not  in 
Hughes’ 
Framework 

Test  &  Evaluation  Master  Plan 
(TEMP) 

P 

Opportunity 

Identification 

CONOPS  Mission  /  Objectives  /  Tasks  / 
Measures  (MOE) 

Related 
Architecture  / 
Acquisitions 
Products 


SV-5,  CV-6 


StdV-1 


CV-2,  CV-4 


SvcV-1,  SvcV-5 


AV-1,  CONOPS 


CONOPS,  OV-1 
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3.  Data  Collection. 


The  research  team  identified  the  desired  interview  candidates  using  the  concept 
of  purposeful  homogenous  sampling  as  discussed  previously.  After  discussion  with 
various  faculty  members  at  the  Air  Force  Institute  of  Technology  and  the  Air  Force 
Center  for  Systems  Engineering,  the  research  team  compiled  a  list  of  potential 
candidates  that  are  actively  involved  in  Air  Force  related  acquisitions.  These 
candidates  shared  qualities  and  attributes  of  having  experience,  expertise,  and 
knowledge  within  their  respective  fields.  The  candidate  list  included  individuals  from 
organizations  throughout  the  Air  Force  at  various  levels  of  command  and  influence 
and  representing  different  geographic  regions  of  the  US.  The  list  also  contained 
individuals  with  varying  career  status  in  the  Air  Force  including  active  duty  military 
and  government  civilians. 

Due  to  time  and  resource  constraints  the  delivery  of  the  interview  itself  was 
conducted  differently  for  participants  not  local  to  the  research  team.  If  the  participant 
was  in  the  local  region,  the  interview  was  typically  conducted  at  the  participant’s  place 
of  work  in  a  private  setting.  If  the  participant  was  not  local,  then  the  interview  was 
conducted  using  a  telephone  conference,  also  in  a  private  setting.  The  questions  and 
structure  were  exactly  the  same  for  both  local  and  non-local  interviews,  and  the  only 
deviation  was  that  the  cards  used  during  the  two  exercises  were  sent  electronically  to 
the  participant  who  then  printed  a  copy  to  reference.  The  local  interviews  were  given 
the  cards  during  the  interview,  which  were  separated.  The  only  limitation  the  research 
team  identified  by  this  change  was  that  the  non-local  participants  might  not  be  able  to 
easily  manipulate  and  organize  the  cards.  This  organization  of  the  cards  would  be 
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useful  when  answering  the  question  regarding  if  the  participant  felt  there  was  any 
ordering  or  grouping  of  the  activities.  However,  the  content  and  cards  themselves, 
separated  or  not,  were  consistent  to  all  participants  and  the  research  team  determined 
that  interviewing  the  appropriate  people  was  worth  this  minor  deviation. 

During  the  interview,  the  research  team  again  explained  the  purpose  of  the 
interview  and  the  general  structure.  The  research  team  told  each  participant  that  the 
interview  was  scheduled  to  last  approximately  30  minutes  with  up  to  an  hour  for 
follow-on  discussion.  The  research  team  also  reminded  participants  that  any  and  all 
comments  would  be  anonymous  and  could  not  be  attributed  to  them.  The  research 
team  did  this  to  avoid  any  participant  skewing  their  responses  or  withholding  any 
negative  information  for  fear  of  reprisal.  Furthermore,  the  research  team  required  data 
based  on  the  respondents  perceptions  and  thoughts,  not  the  “approved”  or  “school¬ 
book”  answers. 

Both  researchers  participated  in  each  interview  regardless  of  the  local  or  non¬ 
local  format.  The  team  used  the  approach  of  tandem  interviewing  as  recommended  by 
Krathwohl  (1998).  This  approach  allows  two  researchers  to  ask  questions  of  the 
participants  during  the  interview.  Krathwohl  claims  that  the  benefits  of  tandem 
interviewing  include  allowing  one  researcher  to  record  responses  while  another  asks 
questions  and  allows  for  further  clarification  if  another  researcher’s  question  requires 
it.  This  approach  served  this  data  collection  well  and  allowed  for  a  more  seamless 
session  with  each  participant.  To  record  each  participant’s  responses  both  researchers 
hand  copied  the  bulk  of  the  answers  and  associated  dialogue.  Conversation  that  was 
unrelated  to  the  discussion  and  purpose  of  the  interview  was  not  recorded.  At  the 


51 


conclusion  of  the  interview  the  research  team  compared  data  for  consistency  and  to 
correct  any  deviations. 

4.  Data  Reduction. 

After  all  the  interviews  and  data  collection  phase  was  finished,  the  research 
team  began  a  process  to  reduce  the  information  from  the  respondents  into  a  more 
understandable  and  useful  form.  All  the  relevant  hand- written  notes  were  transcribed 
using  a  word-processing  tool.  The  researchers  again  compared  notes  and  ensured  that 
any  conflicting  data  was  resolved.  At  this  point,  the  research  team  reduced  the 
remaining  data  using  both  a  quantitative  and  also  a  qualitative  approach.  Regardless 
of  the  approach  used,  the  originating  source  (interviewee)  of  the  data  was  never 
eliminated. 

The  quantitative  data  included  organizing  the  number  of  times  a  particular 
exercise  card  was  selected  during  one  of  the  two  interview  exercises.  For  the  first 
exercise,  the  “As-Is”  process,  the  researchers  recorded  the  number  of  times  each  card 
was  selected  as  well  each  time  it  was  left  out.  The  same  approach  was  used  for  the 
second  exercise,  the  “To-Be”  process.  The  research  team  also  recorded  the  number  of 
times  a  card  was  indicated  for  one  of  the  follow-on  questions  (i.e.  most  important, 
most  time/resource  consuming,  etc.) 

The  researchers  next  evaluated  and  reduced  the  qualitative  comments  given  by 
the  respondents  during  the  interviews.  Due  to  the  open-discussion  format  of  the 
interviews  many  respondents  gave  comments  concerning  a  particular  exercise  card  at 
different  points  throughout  the  interviews.  For  this  reason,  the  research  team  chose  to 
group  all  comments  concerning  a  particular  exercise  card,  rather  than  separate 
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comments  by  exercises  as  in  the  quantitative  reduction  discussed  previously.  The 
research  team  also  grouped  all  responses  given  regarding  maturity  elements  that  were 
missing  or  should  be  added.  Finally,  for  all  noteworthy  comments  not  specific  to  any 
of  the  exercise  cards  or  follow-on  questions,  the  research  team  grouped  these  as 
general  comments. 

5.  Data  Analysis. 

After  organizing  and  grouping  the  comments  and  exercise  card  data,  the 
research  team  began  to  study  and  analyze  the  complex  and  quantitative  and  qualitative 
information.  The  research  team  took  a  two-way  approach  to  the  data  analysis.  The 
first  approach  as  recommended  by  Patton  (2002)  is  deductive  and  seeks  to  analyze  the 
data  through  the  lens  of  an  existing  model,  tool,  or  framework.  This  type  of  analysis 
looks  at  how  a  specific  hypothesis  is  supported  for  or  against  by  the  available  data  and 
commonly  involves  taking  the  responses  given  to  see  how  they  compare  or  contrast  to 
the  concept  in  question  (Patton).  In  reference  to  this  study,  the  hypothesis  used  is  that 
the  Hughes’  Framework  adds  value  to  early  development.  Patton  offers  that 
deductive  analysis  is  primarily  served  by  quantitative  data,  but  qualitative  data  can 
also  be  used.  The  research  team  focused  the  bulk  of  the  validation  effort  on  deductive 
analysis  of  the  interview  data. 

In  contrast  to  deductive  analysis,  inductive  analysis  seeks  to  evaluate  the 
available  data  without  any  lens,  hypothesis,  or  pre-conceived  opinions  (Patton,  2002). 
This  type  of  analysis  is  normally  qualitative  driven  and  attempts  to  look  at  the  data  for 
what  it  is  saying  alone  and  not  in  comparison  to  other  known  ideas.  Taylor  and 
Bogdan  (1984)  mention  that  often  a  research  study  heavy  with  qualitative  data  will 
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start  deductive  and  then  shift  towards  inductive.  This  claim  supports  the  research 
goals  of  this  study.  Besides  the  validation  effort  of  the  framework,  the  researchers 
also  wanted  to  understand  the  larger  context  of  early  development,  acquisitions,  the 
people  involved,  and  any  related  findings  the  interview  results  could  offer.  Inductive 
analysis  is  a  strong  methodology  for  this  type  of  approach  as  Patton  adds,  “the 
researcher  strives  to  look  at  the  data  afresh  for  undiscovered  patterns  and  emergent 
understandings”  (p.454). 

In  support  of  the  inductive  analysis,  the  research  team  used  the  analysis 
techniques  described  by  Patton  (2002)  to  methodically  and  logically  make  sense  of  the 
vast  amount  of  differing  comments  given  by  the  respondents.  Patton  discusses  the 
idea  of  developing  codes  and  categories  to  understand  qualitative  data.  He  uses  the 
concept  of  convergence  to  describe  a  technique  of  looking  at  qualitative  data  for  what 
things  fit  together  and  any  recurring  regularities  (Patton,  2002).  These  regularities 
lend  to  patterns  that  can  be  coded  into  categories  for  further  analysis  (Patton,  2002). 
During  this  coding  process,  Denzin  (1978)  recommends  the  concept  of  investigator 
triangulation  as  a  method  to  strengthen  the  findings  from  a  common  qualitative  data 
source.  The  research  team  adopted  this  approach  and  chose  to  have  both  members 
examine  the  comments  and  come  up  with  a  separate  list  of  recurring  regularities, 
categories,  noteworthy  observations  and  potential  themes.  After  this  initial 
examination  the  team  compared  and  contrasted  lists  as  recommended  by  Denzin. 

6.  Reporting. 

As  part  of  this  validation  effort,  the  research  team  desired  to  present  the  results 
of  this  study  based  on  the  researchers’  opinions  while  including  enough  data  so  that 
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the  reader  can  make  their  own  conclusion  on  the  usefulness  of  the  framework.  As  part 
of  the  data  reduction  effort  the  research  team  grouped  all  exercise  related  comments 
and  presents  these  for  discussion.  The  research  team  leverages  the  use  of  histograms 
heavily  to  show  the  quantitative  data  related  to  the  exercise  cards.  These  histograms 
are  useful,  for  example,  for  demonstrating  the  number  of  times  a  maturity  element  was 
selected  in  the  “As-Is”  or  “To-Be”  processes  or  which  cards  were  rated  more 
important  than  others.  Also  when  appropriate  the  research  team  was  able  to  make 
factual  conclusions  presented  as  ratios  and  percentages  based  on  the  data.  Finally,  the 
research  team  presents  any  derived  themes,  patterns,  lessons-learned,  and  best- 
practices  regarding  this  study  in  summary  format  for  the  reader.  The  results, 
conclusions,  and  recommendations  from  this  study  are  presented  in  the  subsequent 
chapters. 

C.  Application  Methodology 

As  part  of  the  summative  evaluation  research  approach  used  in  the  validation 
effort  of  the  Hughes’  Framework,  the  research  team  examined  if  the  framework  could 
be  applied  to  a  real-world  concept  in  development.  This  approach  would  highlight  not 
only  the  value  of  the  framework,  but  also  the  ease  of  use  in  applying  it.  The  results  of 
this  application  would  be  used  to  support  the  external  validity  or  generalizations  of  the 
findings  discovered  during  the  interviews  to  other  areas  (Krathwohl,  1998).  The 
following  sections  discuss  the  background  and  methodology  of  the  framework  real- 
world  application  effort. 
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1.  Maturity  Assessment  &  Background. 

In  the  summer  of  2010,  the  researeh  team  was  asked  to  provide  an  assessment 
of  the  maturity  of  a  concept  in  the  early  stages  of  development  (i.e.  post-  need 
identification)  for  a  weapon  designed  to  destroy  hard-to-defeat  targets.  The 
sponsoring  military  organization  desired  an  outside  and  unbiased  assessment  of  each 
development  contractor’s  solution  from  an  academic  point  of  view.  The  research  team 
chose  to  use  the  Hughes’  framework  as  a  guide  to  evaluate  each  contractor’s  proposal. 
This  assessment  did  not  aim  to  assess  the  goodness  or  effectiveness  of  a  contractor’s 
approach,  but  would  be  used  to  provide  feedback  on  whether  the  Government  (the 
sponsoring  organization)  has  enough  information  to  determine  the  weapon  concept’s 
maturity  and  preparation  for  further  development. 

For  this  assessment  the  research  team  gathered  data  on  the  currently  available 
information  by  both  the  Government  and  each  contractor.  The  research  team  attended 
developmental  contractor  program  reviews,  researched  background  material,  system 
requirements,  and  other  program  related  documents  to  gain  an  understanding  of  the 
overall  concept  and  each  contractor’s  specific  solution.  The  researchers  provided  an 
assessment  report  to  the  sponsoring  organization  on  information  determined  as 
sufficient,  missing,  or  areas  for  improvement  by  applying  the  Hughes’  Framework. 
Due  to  the  official  or  proprietary  status  in  this  assessment,  no  details  regarding  the 
sponsoring  organization  or  the  weapon  concept  are  provided  in  this  study.  However, 
as  mentioned  previously,  the  researchers  were  seeking  primarily  to  demonstrate  that 
the  Hughes’  Framework  could  be  applied  and  the  level  of  ease/difficulty,  rather  than 
the  added-value  of  applying  it.  This  initial  assessment  effort  highlights  how  the 
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framework  can  be  used  as  a  guide  to  evaluating  a  concept’s  maturity  and  help  identify 


potential  gaps. 

2.  Architecture  Effort. 

After  delivery  of  the  concept  maturity  assessment,  the  sponsoring  organization 
indicated  a  desire  for  the  researchers  to  create  the  maturity  elements  described  as 
missing  or  requiring  improvement.  Although,  the  researchers  had  limited  knowledge 
of  the  program,  they  were  able  to  create  all  the  recommended  maturity  elements  and 
architecture  products  in  a  usable  draft  form.  These  deliverables  were  completed  at 
varying  levels  of  details  based  on  the  available  information.  During  this  effort,  the 
researchers  noticed  that  regardless  of  the  detail  included  in  a  maturity  element, 
applying  the  framework  at  least  highlighted  areas  that  required  more  information.  The 
research  team  used  a  PC-based  application  tool.  System  Architect,  to  create  and  design 
the  architecture  products  and  diagrams  used  in  this  effort.  Table  5,  lists  the  maturity 
elements  and  architecture  products  as  described  in  the  Hughes’  Framework  that  were 
developed  by  the  research  team. 


Table  5  -  Architecture  and  Maturity  Elements  for  Application 


Product 

Code 

Use 

Overview  and  Summary 
Information 

AV-l 

Sets  the  purpose,  scope,  methodology,  products,  and 
expected  analyses  to  be  performed  for  the  entire 
architecture  effort 

Concept  of  Operations 

CONOPs 

Outlines  the  operation  concept  required  by  this  system 

Integrated  Dictionary 

AV-2 

Definition  of  terms 

High-Level  Operational 

Concept  Graphic 

OV-1 

High  level  graphical  and  textual  description  of  operational 
concept 

Operational  Node  Connectivity 
Description 

OV-2 

Connectivity  and  information  flow  between  nodes 
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Product 

Code 

Use 

Organizational  Relationships 
Chart 

OV-4 

Shows  organizational  context,  role  or  other  relationships 
among  organizations  and  units 

Node  Tree 

OV-5a 

Activities,  relationships  among  activities,  inputs  and 
outputs 

Activity  Model 

OV-5b 

Activities,  relationships  among  activities,  inputs  and 
outputs 

Operational  Rules  Model 

OV-6a 

Describes  business  rules  that  constrain  operations 

Systems  Interface  Description 

SV-1 

Identifies  systems,  system  items,  and  their  interconnections 

Due  to  the  official  and  proprietary  status  of  these  products,  the  actual  diagrams  and 
models  are  not  included  in  this  study.  When  finished,  the  researchers  briefed  and 
delivered  these  products  and  the  results  of  this  effort  to  the  sponsoring  organization. 

3.  Feedback. 

The  research  team  requested  feedback  on  the  utility  and  value  of  both  the 
assessment  and  maturity  elements  delivered  to  the  sponsoring  organization.  The 
organization  responded  that  the  assessment  was  very  useful  at  understanding  the 
maturity  of  each  contractor’s  conceptual  solution  and  any  gaps  in  information  that  may 
have  been  overlooked.  The  organization  also  commented  that  the  maturity  elements 
and  architecture  products  delivered  helped  them  understand  the  weapon  concept 
further  and  was  of  value  to  them.  Finally,  the  sponsoring  organization  requested  that 
further  architecture  related  work  and  collaboration  between  future  students  and  faculty 
continue  as  the  development  program  progresses.  In  summary,  this  application  effort 
demonstrates  that  the  Hughes’  Framework  can  be  applied  to  a  real-time  concept  in 
early  development  and  that  in  this  specific  situation  the  results  were  favorable  and 
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perceived  as  adding  value  to  the  overall  development  effort  by  the  sponsoring 
organization. 

D.  Confirmatory  Sources  Methodology 

Related  to  the  application  approach  discussed  previously,  the  research  team 
chose  to  use  the  concept  of  confirmatory  sources  to  support  the  findings  from  the 
interviews  and  demonstrate  the  findings  external  validity.  Patton  (2002)  discusses  the 
concept  of  confirmatory  significance  in  his  research  guidebook,  offering  that  a  finding 
supported  by  another  work  has  confirmatory  significance.  This  concept  led  the 
research  team  to  look  for  approved  works  to  include  approved  government  and 
military  policy,  regulations,  guidance;  as  well  as  any  related  literature  or  other 
scholarly  sources  to  identify  other  works  that  confirm  the  value  or  recommend  using 
of  maturity  elements  within  the  framework  in  early  development.  Although  this 
approach  could  be  confused  with  the  literature  review  that  impacted  creation  of  the 
Hughes’  Framework,  the  researchers  were  searching  for  a  very  narrow  group  of  works 
with  the  criteria  that  they  relate  to  using  maturity  elements  in  early  development  (i.e. 
before  MS-A). 

Development  of  the  framework  by  Hughes  has  much  focus  on  the  notion  that  a 
stage-gated  approach  is  useful.  Since  the  research  team  for  this  study  chose  to  focus 
on  if  the  contents  of  the  framework  add  value  rather  than  the  stage-gated  concept,  the 
search  for  confirmatory  sources  will  only  involve  looking  for  sources  that  recommend 
using  maturity  elements  in  the  framework.  If  the  findings  of  the  interview  results  lead 
towards  a  conclusion  that  the  framework  does  in  fact  add  value  to  assessing  concept 
maturity  during  early  development,  then  the  researchers  will  use  any  potential 
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confirmatory  sources  to  generalize  the  findings  past  the  small  study  sample. 
Regardless  of  the  outcome  of  the  interview  data  leading  to  favorable  or  non-favorable 
recommendation  on  the  framework,  these  confirmatory  sources  should  help  the 
researchers  show  how  the  findings  are  externally  valid. 

E.  Formative  Evaluation 

As  discussed  previously,  the  primary  purpose  behind  this  research  study  is  the 
summative  evaluation  effort  associated  with  validation  of  the  Hughes’  Framework. 
However,  the  research  team  also  took  an  improvement  approach  to  the  framework  as 
discussed  by  Patton  (2002)  regarding  a  formative  evaluation.  A  formative  evaluation 
aims  to  form  or  shape  the  thing  being  studied  with  the  specific  purpose  of 
improvement  (Patton,  2002).  Regardless  of  if  acceptance  of  the  Hughes’  framework 
as  a  tool  for  assessing  concept  maturity  during  early  development  is  validated  or  not, 
the  researchers  will  explore  areas  to  improve  or  build  on  the  framework.  The  research 
and  results  gained  from  the  interviews,  application,  and  confirmatory  sources  methods 
will  all  be  platforms  to  recommend  improvements  to  the  framework  as  well  as  the 
opinions  of  the  researchers. 

F.  Methodology  Summary 

The  overall  methodology  used  in  this  research  is  a  summative  evaluation, 
focused  on  the  validation  effort  of  the  Hughes’  Framework.  The  researchers  also 
looked  to  provide  a  formative  evaluation  to  recommend  any  improvements  to  the 
framework  if  deemed  appropriate.  The  concept  of  triangulation  as  a  method  to 
strengthen  the  findings  and  rigor  of  this  study  are  leveraged  significantly  throughout 
the  validation  effort.  The  validation  effort  is  separated  into  three  sections  (1) 
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interviews  with  relevant  personnel,  (2)  an  application  of  the  framework  as  tool  to  a 
DoD  concept  in  real-time,  (3)  and  an  evaluation  of  accepted  policies,  best-practices, 
and  guidance  as  they  compare  to  the  framework.  During  this  effort,  the  research  team 
looked  to  find  the  interview  portion  of  this  study  internally  valid,  and  then  use  the 
application  and  confirmatory  sources  portions  to  expound  on  the  findings  external 
validity. 

The  interviews,  as  the  primary  vehicle  for  validation,  are  focused  on  personnel 
within  the  DoD  who  have  experience  in  early  development  and  can  best  provide 
insight  into  the  frameworks  potential  value.  The  researchers  broke  down  the 
framework  into  its  maturity  elements  and  while  applying  preference  measurement  and 
conjoint  analysis  concepts  asked  the  interviewees  for  comments  related  to  the  value 
and  issues  for  each  element.  The  application  effort,  led  the  researchers  to  apply  the 
framework  as  a  tool  to  assess  a  concept  in  development’s  maturity  and  then 
successfully  create  the  missing  maturity  elements  in  a  draft  form  or  highlight  needed 
information.  Finally,  the  confirmatory  sources  methodology  involves  the  researchers 
evaluating  accepted  guidance  on  early  development  and  comparing  them  to  the 
framework  to  look  for  similarities  and  disconnects. 
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IV.  Analysis  and  results 


The  triangulation  method  strengthens  this  analysis  by  combining  different 
research  techniques  independently.  The  first  research  technique  used  to  validate  the 
Hughes’  Framework  were  structured  interviews  with  knowledgeable  individuals 
experienced  in  early  concept  development.  The  second  research  technique  is  an 
attempt  to  apply  the  framework  to  a  real-world  concept  that  is  currently  in  the  early 
stages  of  development.  The  last  research  technique  is  an  analysis  of  approved  policy 
and  guidance  that  relates  to  the  maturity  elements  of  the  framework. 

Structured  Interviews 

The  interview  was  framed  to  gather  the  perceptions  and  viewpoints  of  the 
value  of  the  maturity  elements  within  the  framework  as  well  as  any  relevant  discussion 
on  related  topics  such  as  early  development  and  acquisitions.  The  tables  in  this 
chapter  are  an  aggregation  of  the  comments  made  by  the  respondents  about  cards  that 
were  presented  to  them  as  well  as  any  other  thoughts  the  respondents  had  in  general 
about  the  acquisition  process.  These  cards  represented  maturity  elements  within  the 
framework  as  well  as  some  other  elements  to  help  gauge  any  other  maturity  elements 
that  may  be  missing. 

Tables  near  the  end  of  the  structured  interview  section  are  compilations  or  the 
respondents’  answers  to  the  questions  of  “what  maturity  elements  are  used  currently, 
what  maturity  elements  would  be  important  to  you  if  you  were  making  the  decision, 
what  would  be  your  top  three  choices,  and  a  notional  order  you  think  they  should 
occur.” 
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Exercise  Card  “A”  Results. 


The  comments  about  this  card  were  wide  ranging,  see  Table  6  below.  Some 
thought  the  acquisition  community  did  a  good  job  at  this,  while  others  felt  the  exact 
opposite.  It  was  a  common  feeling  of  the  interviewees  that  the  acquisition  community 
does  not  accurately  capture  the  cost  impact  that  concept  solutions  may  have  on 
external  systems.  Also  many  interviewees  thought  that  this  is  an  area  that  needs  more 
attention  and  resources  since  cost  overruns  and  schedule  delays  are  far  too  prevalent. 
Some  comments  on  how  to  improve  this  area  included  increasing  the  breadth  of 
experience  of  the  cost  analysts  involved  with  early  cost  estimates  and  more  resources 
for  training. 
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Table  6  -  Qualitative  Comments  for  Exercise  Card  “A” 


Rough  order  of  magnitude  initial  cost  and  schedule  estimates. 

Current 

Important 

Top  three 

Comments  by  interviewee  #  X  on  Maturity  Element  A 

X 

X 

1.  One  of  the  most  time  intensive  tasks.  Data  trickles  into  the  costing  teams  and  is  heavily 
dependent  on  external  sources.  Doing  costing  for  new  alternatives  can  take  a  lot  of  time. 

X 

2.  This  is  done  sometimes  because  OSD  demands  it.  More  emphasis  should  be  placed  on 
this. 

X 

X 

X 

3.  Spot  on. 

X 

X 

4.  This  happens  near  the  end. 

X 

X 

X 

X 

6.  This  is  refined  during  the  AoA  and  is  done  well. 

X 

X 

7.  This  is  very  optimistic  in  its  projections  for  cost  and  schedule  (CAIG  &  OSD)  are 
funding  to  an  80%  confidence  level.  Cost  estimates  are  hard  to  do,  but  basing  it  on  a 
previous  related  system  might  not  be  the  best  approach. 

X 

X 

8.  This  is  not  done  accurately  and  fails  to  capture  the  possible  impact  on  other  systems. 

Lots  of  from  the  hip,  big  guesses,  only  recently  bringing  real  cost  analysts  to  help  estimate 
this  we  are  bad  at  looking  how  the  cost  applies  across  many  stakeholders  and  systems. 

X 

X 

X 

9.  We  can  do  better  when  the  costs  impact  other  systems,  the  hidden  costs  to  the  system. 

X 

10.  We  try  to  do  this  but  it  needs  improvement.  The  people  that  do  this  work  are  licking 
the  breadth  of  experience  needed  especially  since  this  is  hard  to  do  early  on.  Lots  of  focus 
on  the  cost  of  the  concept,  need  to  look  at  how  it  affects  other  systems  and  the  lifecycle 
(maintenance,  logistics,  ALC,  operational)  cost. 

X 

X 

X 

11.  Threats  to  hold  to  MSA  budget  projections  and  threaten  a  Nunn  McCurdy  breach  are 
common  enough  that  more  emphasis  should  be  placed  on  cost  then  the  technical  accuracy. 
It’s  really  hard  to  estimate  early  on  but  we  often  put  more  emphasis  on  cost  and  schedule 
than  the  technical  piece. 

When  asked  “What  is  important?”  when  presented  with  all  of  the  cards  it  was  tied  for 
third  with  nine  out  of  then  ten  interviewees  that  responded  selecting  this  card.  When 
limited  to  only  their  top  three  choices  it  was  tied  for  fourth  with  only  three  out  of  the 
ten  interviewees  that  responded  choosing  this  card.  When  the  interviewees  were  asked 
to  order  the  cards  by  when  they  believed  the  activities  should  occur  it  was  eleventh  out 
of  sixteen.  After  analyzing  the  comments  and  responses  to  the  questions  asked  the 
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research  team  can  conclude  that  the  maturity  elements  in  the  framework  represented 


by  this  card  does  add  value  to  the  decision  making  process. 

Exercise  Card  “B”  Results. 

Overall  the  comments  about  the  activities  represented  in  this  card  were 
favorable,  see  Table  7  below.  The  overall  feeling  is  that  although  these  activities  can 
be  improved,  they  are  being  accomplished  better  than  in  the  past.  One  noteworthy 
comment  is  that  more  emphasis  should  be  placed  on  the  timing  and  sequence  of  the 
interfaces,  rather  than  only  a  description  of  what  the  interfaces  are. 


Table  7  -  Qualitative  Comments  for  Exercise  Card  “B” 


Description  of  interfaces  between  system  elements  and  the  resources  that  flow 
between  them  as  well  as  the  operational  activities  they  support. 

Current 

Important 

Top  three 

Comments  by  interviewee  #  X  on  Maturity  Element  B 

X 

X 

X 

X 

4.  Once  an  OV-4  is  done  we  can  map  out  the  interfaces. 

X 

X 

X 

X 

X 

6.  This  is  done  better  than  we  ever  have  because  we  are  mostly  involved  with 
system  of  systems  and  computer  based  developments. 

X 

X 

7.  This  is  currently  just  a  high  level  description,  not  as  detailed  as  it  can  be. 

X 

X 

8.  When  “Architectures”  came  into  being  they  lent  themselves  very  well  to 
describing  interfaces,  but  still  needs  to  be  done  better.  More  emphasis  should  be 
placed  on  timing  and  sequence,  not  just  what  the  interfaces  are.  It’s  not  just  wires 
to  wires,  its  integration  between  systems  that’s  very  complex. 

X 

X 

9.  System  of  systems  can  be  a  challenge  especially  with  net  ready  KPPs. 

X 

10.  A  lot  of  the  requirements  are  done  better  because  they  are  connected  to  the 
operators. 

X 

11.  Traditionally  not  done  prior  to  MSA  and  is  dependent  upon  the  contractors 
that  are  proposing  the  solution. 
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When  asked  “What  is  important?”  when  presented  with  all  of  the  eards  it  was  tied  for 
seventh  with  seven  out  of  then  ten  interviewees  that  responded  choosing  this  card. 
When  limited  to  only  their  top  three  choices  it  was  tied  for  ninth  with  only  one  out  of 
the  ten  interviewees  that  responded  selecting  this  card.  When  the  interviewees  were 
asked  to  order  the  cards  by  when  they  believed  the  activities  should  occur  it  was  sixth 
out  of  sixteen. 

After  listening  to  the  interviewees’  description  of  the  current  acquisition 
process  they  nearly  all  said  that  this  card  was  part  of  it.  Most  respondents  included 
this  card  in  their  preferred  “should  be”  process.  Although  few  respondents  included 
the  associated  activities  in  their  top-three  most  essential,  the  majority  indicated  these 
activities  add  value  to  decision  makers. 

Exercise  Card  “C”  Results, 

The  comments  about  exercise  card  “C”  were  wide  ranging,  see  Table  8.  Some 
respondents  thought  we  did  a  good  job  at  this  while  others  felt  the  exact  opposite.  In 
general  more  effort  and  consideration  should  be  put  into  understanding  the 
assumptions  and  constraints  that  will  constrain  the  solution  by  bringing  in  the  users 
earlier  and  continue  getting  feedback  during  the  process.  Some  respondents 
mentioned  that  too  often  a  preferred  solution  has  already  been  decided  upon  and  this  is 
used  as  a  tool  to  artificially  steer  the  selection  process  to  that  preferred  solution.  Other 
noteworthy  comments  included  that  information  technology  constraints  are  not 
understood  or  considered  as  much  as  they  should  be. 
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Table  8  -  Qualitative  Comments  for  Exercise  Card  “C” 


List  of  assumptions,  constraints  and  enabling  capabilities  that  are  required  for 

the  system  to  operate  effectively. 

Current 

Important 

Top  three 

Comments  by  interviewee  #  X  on  Maturity  Element  C 

X 

X 

1.  More  emphasis  should  be  placed  on  this. 

X 

X 

X 

X 

X 

X 

X 

4.  This  is  done  fairly  well  (Identification  of  the  scope  of  the  problem) 

X 

X 

X 

X 

X 

6.  This  is  not  done  well  because  often  the  solution  is  already  defined. 

X 

7.  Helps  you  understand  the  system.  Not  enough  technical  detail  this  early  on  to 
develop  these  architecture  products. 

X 

X 

8.  Systems  of  systems  do  not  understand  interfaces  and  architectures  are  not 
dynamic.  We  need  more  user  involvement  and  more  feedback  during  the  process. 

X 

X 

9.  Is  tied  to  the  tasks,  conditions  and  standards  you  get  from  the  operations  guys 
during  the  CBA.  Requirements  elicitation  may  miss  some  assumptions  & 
constraints.  Future  political  impacts  and  leadership  are  hard  to  think  about  early 
on,  but  can  greatly  affect  a  program. 

X 

10.  We  are  not  considering  Information  Technology  up  front  as  much  as  it  needs 
to  be. 

X 

X 

X 

11.  For  the  study  plan  they  come  up  with  the  Ground  Rules  and  Assumptions 
(GRA),  this  is  done  adequately. 

When  asked  “What  is  important?”  when  presented  with  all  of  the  cards  it  was  tied  for 
first  with  all  ten  interviewees  that  responded  picking  this  card.  When  limited  to  only 
their  top  three  choices  it  was  tied  for  second  with  only  four  out  of  the  ten  interviewees 
that  responded  choosing  this  card.  When  the  interviewees  were  asked  to  order  the 
cards  by  when  they  believed  the  activities  should  occur  it  was  second  out  of  sixteen. 

This  card  ranked  high  in  the  current  process  and  what  the  respondent  thought 
the  process  should  be.  This  card  also  ranked  very  high  in  the  order  in  which  it  should 
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happen.  Based  on  the  comments  and  responses,  the  research  team  concludes  that  there 
is  value  in  the  information  elements  this  card  represents. 

Exercise  Card  “D”  Results. 

The  majority  of  interviewees  said  that  understanding  the  organizational 
relationships  and  flow  of  resources  is  currently  accomplished  poorly  during  early 
development,  see  Table  9.  They  see  what  is  done  now  as  just  a  list  of  stakeholders  but 
it  needs  to  be  more  than  that.  A  standardized  process  to  stakeholder  analysis  during 
early  development  was  suggested  by  a  couple  individuals  as  a  way  to  improve  this 
activity.  Interestingly,  even  though  many  said  this  was  important  and  should  be  done 
early,  when  asked  to  order  the  cards  by  when  they  believed  this  activity  should  occur 
the  majority  placed  this  card  towards  the  end  of  the  process. 
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Table  9  -  Qualitative  Comments  for  Exercise  Card  “D” 


Organizational  relationships,  their  roles  and  responsibilities,  and  how  the  flows  of 

resources  are  expected  to  occur. 

Current 

Important 

Top  three 

Comments  by  interviewee  #  X  on  Maturity  Element  D 

X 

X 

X 

X 

4.  From  a  C4ISR  perspective  interface  issues  are  better  handled  by  contractors. 

X 

X 

X 

X 

6.  Is  “Goobered  up.”  This  is  better  when  it  addresses  cross  functional 
relationships,  but  it  could  use  more  up-front  resources.  It’s  hard  to  get  money  for 
programs  that  don’t  exist  yet  early  on. 

X 

7.  This  helps  gather  the  environmental  context.  Not  enough  technical  detail  this 
early  on  to  develop  these  architecture  products. 

X 

X 

8.  We  don’t  have  the  full  level  of  detail  on  this.  Stakeholder  analysis  is  not  done 
well,  but  it  is  very  specialized.  Time  should  be  taken  upfront  in  a  disciplined 
process  to  perform  this  analysis.  We  don’t  have  a  disciplined  process  to  do  this  in 
the  Air  Force;  it’s  an  important  but  difficult  task. 

X 

X 

9.  Missing  external  areas  but  are  getting  better.  We  are  starting  to  include  the 
sustainment  stakeholders. 

X 

10.  This  has  improved  mainly  because  we  have  tried  to  standardize  the  process; 
however  standardizing  can  add  complexity,  time  and  cost  to  the  process. 

X 

X 

11.  Still  being  figured  out  how  to  formalize  this  proeess.  Shouldn’t  just  be  a  list  of 
stakeholders,  we  need  to  get  all  the  stakeholders  involved. 

When  asked  “What  is  important?”  when  presented  with  all  of  the  cards  it  was  tied  for 
seventh  with  seven  out  of  the  ten  interviewees  that  responded  choosing  this  card. 
When  limited  to  only  their  top  three  choices  it  was  tied  for  last  with  no  one  choosing 
this  card.  When  the  interviewees  were  asked  to  order  the  cards  by  when  they  believed 
the  activities  should  occur  it  was  thirteenth  out  of  sixteen. 
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This  card  was  said  to  be  important  by  nearly  all  interviewees  so  it  can  be 
reasoned  that  it  adds  value.  However,  the  timing  at  whieh  the  respondents  indicated 
that  they  wanted  this  activity  accomplished  was  later  than  what  the  Hughes’ 
Framework  reeommends.  This  delay  may  be  due  to  the  different  policy  guidance 
regarding  this  subject.  The  resources  and  initial  identification  of  the  organizations 
responsible  for  developing  the  solution  are  what  most  DoD  poliey  foeuses  on.  The 
respondents  were  focused  mainly  on  the  organizations  that  would  be  involved  in 
supporting  the  solution  and  their  general  feeling  was  that  supporting  organizations 
would  not  be  identified  until  the  system  is  defined.  These  reasons  may  explain  the 
later  sequencing  for  this  card  by  the  interviewees. 

Exercise  Card  “E”  Results, 

The  general  feeling  about  card  “E”  is  that  is  it  done  adequately  for  the  most 
part  but  it  can  be  performed  better,  see  Table  10.  Respondents  mentioned  that 
consistency  can  be  a  problem  with  the  decomposition  leading  to  missing 
considerations  such  as  safety  and  security.  Also,  they  claimed  that  this  activity 
focuses  too  narrowly  on  the  funetional  aspeets  of  the  solution,  instead  of  supporting  or 
external  tasks.  Furthermore,  respondents  worried  that  a  prescribed  solution  may 
influenee  this  decomposition  if  a  rigorous  approach  is  not  adhered  to. 
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Table  10  -  Qualitative  Comments  for  Exercise  Card  “E” 


A  decomposition  of  all  operational  activities  or  tasks,  and  the  inputs/outputs 

between  them. 

Current 

Important 

Top  three 

Comments  by  interviewee  #  X  on  Maturity  Element  E 

X 

X 

X 

X 

X 

4.  We  try  to  do  this  but  it  is  not  consistent. 

X 

X 

X 

X 

X 

6.  This  is  done  the  worst  but  it  is  very  important.  This  is  part  of  the  CBA. 
Operational  Viewpoints  capture  this  data.  We  do  it,  but  could  do  a  much  better 
job. 

X 

7.  Architecture  products  assist  with  this.  The  system  is  not  defined  as  well  as  it 
could  be. 

X 

8.  Typically  people  handle  requirements  in  this  way  but  it  prescribes  the  solution. 
Other  areas  typically  not  involved  that  should  be  are  safety  and  security.  A 
successful  (but  not  done  in  the  DoD)  method  is  to  hire  requirements  analysts.  A 
lot  of  miscommunication  on  what  “decomposition”  really  means.  We  put  the 
solution  into  the  decomposition  too  early  and  we  only  do  it  from  a  function  view. 

We  need  to  go  farther  than  just  the  functions  for  the  system. 

X 

X 

9.  The  systems  engineering  approach  takes  this  into  account. 

X 

10.  Starting  to  be  fleshed  out,  at  the  early  stages  we  are  starting  to  decompose  the 
concept  and  operational  activities,  can  improve  though. 

X 

X 

11.  This  comes  out  of  the  CBA  and  gives  a  good  idea  of  the  mission  areas.  This 
can  be  done  better  but  the  increases  will  not  impact  the  decision. 

When  asked  “What  is  important?”  when  presented  with  all  of  the  cards  it  was  tied  for 
seventh  with  seven  out  of  the  ten  interviewees  that  responded  choosing  this  card. 
When  limited  to  only  their  top  three  choices  it  was  tied  for  seventh  with  only  two  out 
of  ten  choosing  this  card.  When  the  interviewees  were  asked  to  order  the  cards  by 
when  they  believed  the  activities  should  occur  it  was  fifth  out  of  sixteen. 
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The  information  captured  in  this  card  was  regarded  highly  based  on  the  value 
that  it  could  provide  to  decision  makers  if  it  is  done  properly.  This  is  a  product  of  the 
CBA  process  and  is  a  great  way  to  define  the  system  if  the  decomposition  is  performed 
to  fulfill  the  requirements  of  the  need  versus  decomposing  to  a  prescribed  solution. 

Exercise  Card  “F”  Results, 

There  is  a  difference  of  opinion  on  how  satisfactorily  the  activities  associated 
with  card  “F”  are  performed  currently,  see  Table  11.  This  difference  may  be  due  to 
the  interviewees’  interpretation  of  the  card  and  that  they  may  have  only  commented  on 
one  aspect  of  the  information  contained  within.  The  research  team  attempted  to  clarify 
the  meaning  of  the  card  to  the  interviewees  but  a  couple  respondents  kept  to  their 
initial  responses.  Architecture  products  were  mentioned  as  a  way  to  capture  and 
define  the  information  in  this  card  but  some  worried  that  only  a  portion  of  what  needs 
to  be  documented  is  being  addressed  early  in  the  process.  A  few  of  the  interviewees 
noted  that  a  majority  of  the  information  is  just  not  available  early  on  to  capture  these 
aspects  of  a  concept.  This  lack  of  available  information  is  a  likely  reason  for  the 
respondents’  comments  concerning  the  inadequate  completion  of  card  “F’s”  activities. 
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Table  1 1  -  Qualitative  Comments  for  Exercise  Card  “F” 


Identification  of  the  system  as  a  whole,  the  elements  that  make  up  the  system  and 
their  interconnections  internally  and  external  to  the  systems. 

Current 

Important 

Top  three 

Comments  by  interviewee  #  X  on  Maturity  Element  F 

X 

X 

1.  This  is  done  at  a  low  level  of  quality  and  a  lot  of  people  just  think  this  is  just  an 
OV-1. 

X 

3.  Done  fairly  well. 

X 

X 

4.  The  desired  system. 

X 

X 

X 

X 

6.  The  Work  Breakdown  Structures  are  done  well 

X 

X 

X 

7.  Helps  us  understand  the  high  level  concept.  We  are  getting  better  at  developing 
architecture  products,  maybe  some  OV-ls  are  done  early  on  now  but  most  are  put 
off  until  MSB.  It  would  help  to  have  a  plan  to  flesh  out. 

X 

8.  This  is  done  well  once  the  contractors  have  responsibility,  but  it  is  not  done 
well  or  at  all  before  then. 

X 

9.  This  is  core  to  acquisition  planning.  Interfaces  for  major  efforts  are  done  well 
but  funding  to  do  this  work  is  an  issue. 

X 

10.  Close  to  (B)  done  a  little  better  since  it  is  close  to  the  concept. 

X 

X 

11.  The  CCTD  is  supposed  to  address  this;  maybe  it  is  too  specific  for  being  so 
early.  This  is  done  during  the  analysis  of  alternatives  and  can  be  done  better. 

When  asked  “What  is  important?”  when  presented  with  all  of  the  cards  it  was  tied  for 
seventh  with  seven  out  of  the  ten  interviewees  that  responded  selecting  this  card. 
When  limited  to  only  their  top  three  choices  it  was  tied  for  ninth  with  only  one  out  of 
the  ten  interviewees  that  responded  choosing  this  card.  When  the  interviewees  were 
asked  to  order  the  cards  by  when  they  believed  the  activities  should  occur  it  tied  for 
eighth  out  of  sixteen. 

The  value  in  this  card  is  the  description  of  the  system.  The  information 
elements  that  this  card  represents  are  the  concept  solution  and  all  of  its 
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interconnections.  This  information  provides  context  to  the  decision  maker  on  how  the 
system  operates  and  impacts  other  systems. 

Exercise  Card  “G”  Results, 

Overall  the  interviewees  claimed  that  the  information  captured  in  card  “G”  is 
created  adequately,  but  it  could  be  improved  and  accomplished  earlier  if  funding  and 
better  management  is  available,  see  Table  12.  One  comment  of  note  is  that  acquisition 
professionals  spend  a  significant  time  on  “what  we  want  to  do,”  and  this  element  is 
done  well,  but  the  “who  do  you  want  to  do  it”  is  lacking.  Some  respondents 
mentioned  that  a  possible  reason  for  this  inadequacy  is  that  personnel  analysts  are  no 
longer  around. 
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Table  12  -  Qualitative  Comments  for  Exercise  Card  “G” 


A  description  of  the  functions  performed  by  systems  and  the 
interactions/resource  flows  required  to  perform  that  function. 

Current 

Important 

Top  three 

Comments  by  interviewee  #  X  on  Maturity  Element  G 

X 

X 

X 

4.  This  is  done  decently.  Early  on  it  is  solutions  based  versus  capabilities  based. 

X 

X 

X 

X 

6.  Use  to  be  important  now  mixed  in  with  the  capability  based  assessment. 

X 

8.  This  is  not  done  upfront  and  needs  funding  and  better  management. 

X 

X 

9.  We  do  a  good  job  with  architecture  and  mission  simulation.  Problem  is 
architecture  guys  think  good  systems  engineering  will  figure  it  out.  Doing  these 
documents  after  the  fact  is  what  usually  happens. 

X 

10.  Can  put  a  lot  of  time  into  creating  but  the  “What  you  want  to  do”  is  done  well 
but  the  “Who  you  want  to  do  it”  is  lacking.  Personnel  analysts  are  no  longer 
around  so  who  are  you  going  to  get  to  do  it? 

X 

When  asked  “What  is  important?”  when  presented  with  all  of  the  cards  it  was  tied  for 
thirteenth  with  five  out  of  the  ten  interviewees  that  responded  choosing  this  card. 
When  limited  to  only  their  top  three  choices  it  was  tied  for  last  with  no  respondents 
selecting  this  card.  When  the  interviewees  were  asked  to  order  the  cards  by  when  they 
believed  the  activities  should  occur  it  was  twelfth  out  of  sixteen. 

Changes  in  policy  may  have  diminished  the  importance  of  this  maturity 
element  but  five  out  of  ten  respondents  said  it  was  important  just  not  a  top  priority  for 
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them  if  they  were  making  the  decision.  The  researchers  conclude  that  the  activities 
associated  with  card  “G”  can  add  value  to  decision  makers. 

Exercise  Card  “H”  Results, 

The  observations  regarding  card  “H”  are  depicted  in  Table  13  with  many 
respondents  commenting  on  its  difficulty.  The  comment  that  this  is  hard  to 
accomplish  early  on  prior  to  MSA  was  mentioned  multiple  times.  This  factor  may  be 
due  to  the  reasons  mentioned  previously  that  the  level  of  detailed  information  is  just 
not  available  during  the  early  parts  of  development.  Respondents  offered  that  these 
activities  should  be  developed  with  the  users  early  to  help  address  cost  effectiveness, 
otherwise  a  presupposed  solution  may  surface  that  is  neither  useful  nor  meet  the  user’s 
needs. 
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Table  13  -  Qualitative  Comments  for  Exercise  Card  “H” 


A  mapping  of  desired  system  capabilities  to  the  operational  activities  that  must  be 
performed  and  the  system  functions  that  support  them. 

Current 

Important 

Top  three 

Comments  by  interviewee  #  X  on  Maturity  Element  H 

X 

X 

X 

1.  One  of  the  more  time  intensive  tasks.  Prevents  developing  something  that  is  not 
useful,  or  a  solution  that  does  not  actually  do  what  the  user  needs  (no  presupposed 
solution.) 

X 

X 

4.  Is  still  growing  pre-MSA. 

X 

X 

X 

X 

6.  This  is  done  in  the  decomposition. 

X 

X 

X 

8.  One  of  the  deliverables  is  a  traceability  document.  We  do  a  decent  job  of  this 
mapping. 

X 

X 

9.  Requirements  traceability  is  not  done  rigorously  early  on,  but  we  are  improving. 

X 

10.  It  is  hard  to  do,  and  it  is  trying  to  be  done. 

X 

X 

11.  The  user  has  to  do  this  early  on  since  cost  effectiveness  is  a  big  issue  (required 
Vs  nice  to  have).  It  is  hard  to  do  this  Pre-MSA. 

When  asked  “What  is  important?”  when  presented  with  all  of  the  cards  it  was  tied  for 
tenth  with  six  out  of  the  ten  interviewees  that  responded  choosing  this  card.  When 
limited  to  only  their  top  three  choices  it  was  tied  for  ninth  with  only  one  respondent 
choosing  this  card.  When  the  interviewees  were  asked  to  order  the  cards  by  when  they 
believed  the  activities  should  occur  it  was  tenth  out  of  sixteen. 

This  card  ranked  relatively  low  for  importance  and  how  early  it  should  be 
done.  It  was  also  commented  that  these  activities  can  require  a  significant  amount  of 
time  and  effort  to  produce.  The  maturity  elements  represented  in  this  card  may 
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provide  some  value  to  the  decision  maker  but  the  time  and  resources  needed  to 
produce  it  may  not  be  worth  it.  In  a  constrained  time  and  budget  environment  at  least 
in  these  early  stages  of  development  resources  could  be  allocated  to  other  areas  that 
may  have  a  more  meaningful  impact  on  the  decision  maker’s  choice.  More  research  in 
the  area  of  research  allocation  may  be  useful  to  determine  the  appropriate  level  of 
effort. 

Exercise  Card  “I”  Results, 

A  common  response  to  card  “I”  was  that  developers  are  often  too  optimistic 
with  technology  projections  on  when  they  are  ready  to  be  incorporated  into  system 
developments.  This  lofty  optimism  can  cause  delays  and  expensive  cost  overruns 
while  waiting  for  the  technology  to  mature,  see  Table  14.  Technology  roadmaps, 
TRLs,  TDSs  were  brought  up  as  ways  to  ensure  that  technologies  are  ready.  However, 
if  the  technologies  are  not  mature  at  specific  development  point,  respondents  said  we 
need  a  better  idea  of  what  “off-ramps”  are  available.  Better  cooperation  with  industry 
and  incorporating  manufacturing  readiness  levels  early  can  help  us  understand  what  is 
feasible. 
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Table  14  -  Qualitative  Comments  for  Exercise  Card  “I” 


A  listing  of  maturing  technologies  that  the  development  of  the  system  may  he 

dependent  upon. 

Current 

Important 

Top  three 

Comments  by  interviewee  #  X  on  Maturity  Element  I 

X 

X 

X 

X 

X 

4.  Is  still  growing  pre-MSA. 

X 

X 

X 

X 

X 

6.  This  is  done  well  with  Technology  Readiness  Levels  and  Technology  Readiness 
Assessments,  but  they  are  overly  optimistic  with  their  maturation  plans.  We  need 
better  off-ramps  and  to  be  realistic  with  technologies  that  are  not  mature. 

X 

X 

X 

7.  This  is  important  if  it  is  a  technologically  intensive  system.  This  is  usually 
obvious  but  contains  a  lot  of  optimism. 

X 

X 

8.  Technology  transition  needs  better  cooperation  with  industry,  immature 
technologies  and  the  people  working  them  can  be  a  problem.  Maturing 
technologies  should  be  tied  with  developments. 

X 

X 

9.  There  is  a  need  for  technology  roadmaps,  however,  there  are  cultural  difference 
between  what  the  labs  determine  is  mature  and  what  the  acquisition  needs.  What 
technologies  that  are  needed  in  the  future  should  be  tied  into  development 
planning.  Acceptance  of  what  a  TRL  really  is  can  be  debated. 

X 

10.  Needs  to  be  done  more.  We  heavily  rely  on  Technology  readiness  levels,  but 
we  can  use  manufacturing  readiness  levels  in  our  maturity  assessments.  We  should 
try  and  get  the  MRL  people  up  front  to  understand  what’s  feasible. 

X 

X 

11.  Technology  development  strategy  and  critical  technology  identification  is 
done  well  currently  when  Lab  representatives  have  input. 

When  asked  “What  is  important?”  when  presented  with  all  of  the  cards  it  was  tied  for 
fourth  with  eight  out  of  the  ten  interviewees  that  responded  choosing  this  card.  When 
limited  to  only  their  top  three  choices  it  was  tied  for  third  with  only  three  out  of  the  ten 
interviewees  that  responded  picking  this  card.  When  the  interviewees  were  asked  to 
order  the  cards  by  when  they  believed  the  activities  should  occur  it  was  seventh  out  of 
sixteen. 
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Although  there  was  some  debate  on  the  accuracy  of  the  technology 
assessments,  there  was  no  debate  that  developers  need  to  understand  the  level  of 
technological  dependencies  each  concept  may  have.  Knowing  the  risks  that  immature 
technologies  may  introduce  to  a  solution  can  make  a  huge  impact  on  the  decision 
maker’s  choice.  The  researchers  assess  that  the  added  value  of  the  maturity  elements 
represented  by  this  card  is  high. 

Exercise  Card  “J”  Results, 

Card  “J”  was  not  a  part  of  the  Hughes’  Framework  but  was  added  in  the 
interview  to  get  opinions  on  its  value  early  on  in  the  development  process  and  to 
understand  reasons  for  its  exclusion  from  the  framework.  The  general  consensus  was 
that  a  listing  of  the  standards  is  beneficial,  but  it  is  typically  assumed  that  any  system 
in  development  will  be  in  compliance  with  accepted  standards,  see  Table  15.  The  time 
and  effort  that  would  be  used  to  understand  these  standards  early  in  development 
would  likely  not  be  justified. 
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Table  15  -  Qualitative  Comments  for  Exercise  Card  “J” 


A  listing  of  all  industry,  national  and  international  standards  that  apply  to  the 

system. 

Current 

Important 

Top  three 

Comments  by  interviewee  #  X  on  Maturity  Element  J 

X 

X 

X 

4.  The  desired  system. 

X 

X 

6.  We  do  a  decent  job.  However,  when  MIL-SPEC  went  away  but  ISO  standards 
did  not  pick  up  everything. 

X 

7.  There  is  not  a  lot  of  emphasis  at  MSA,  only  touched  briefly  and  put  off  until 
MSB. 

X 

8.  NESIE,  we  are  more  and  more  aware  of  the  standards  and  paying  more 
attention,  but  could  do  better. 

X 

X 

9.  Air  worthiness  and  air  certification.  This  is  pretty  straightforward  to  use  but 
lots  of  challenges  to  get  agreement  on  standards. 

X 

10.  We  can  do  this  better,  yes  we  probably  should  do  it  better,  but  who  is  going  to 
pay  for  it? 

X 

11.  Obvious  ones  are  identified  post  AoA  in  the  systems  viewpoints,  but  they  are 
not  a  discriminator  when  choosing  between  concepts. 

When  asked  “What  is  important?”  when  presented  with  all  of  the  cards  it  was  last  with 
only  three  out  of  the  ten  interviewees  that  responded  choosing  this  card.  When  limited 
to  only  their  top  three  choices  it  was  tied  for  last  with  no  interviewees  indicating  this 
card  was  most  important.  When  the  interviewees  were  asked  to  order  the  cards  by 
when  they  believed  the  activities  should  occur  it  was  sixteenth  out  of  sixteen.  These 
results  support  the  authors’  of  the  framework  intention  for  only  including  maturity 
elements  that  add  more  value  compared  to  their  expected  cost  and  are  most  appropriate 
during  early  development. 
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Exercise  Card  “K”  Results, 


The  general  responses  for  card  “K”  were  that  the  described  activities  are 
currently  accomplished  to  a  certain  degree,  but  they  should  be  done  better  and  earlier, 
see  Table  16.  Some  respondents  proposed  that  a  way  to  improve  these  activities  was 
with  a  more  uniform  process  to  capture  and  understand  all  of  the  capabilities. 
Respondents  further  commented  that  although  combining  capabilities  to  solve  a 
problem  can  enhance  the  process;  if  new  capabilities  are  discovered  after  development 
has  begun,  then  integration  can  become  a  problem. 


Table  16  -  Qualitative  Comments  for  Exercise  Card  “K” 


A  listing  of  capabilities  needed  to  solve  the  problem  decomposed  down  to  lower 
level  enabling  capabilities  and  the  dependencies  between  them. 

Current 

Important 

Top  three 

Comments  by  interviewee  #  X  on  Maturity  Element  K 

X 

X 

X 

X 

X 

X 

X 

X 

4.  What  do  we  need  to  fix?  Government  does  a  pretty  good  job,  but  this  needs 
better  documentation. 

X 

X 

X 

6.  This  is  done  well.  We  establish  this  early  and  often. 

7.  Not  enough  technical  detail  this  early  on  to  develop  these  architecture  products. 

X 

X 

X 

8.  This  is  done  okay,  but  is  not  done  early  enough. 

X 

X 

9.  From  a  requirements  side  we  can  do  a  better  job  but  we  need  a  more  uniform 
process  to  capture  and  to  understand  all  the  capabilities. 

X 

10.  We  are  doing  this  well,  and  trying  to  do  it  better.  But  do  we  look  at  the 
combination  of  capabilities  to  solve  the  problem? 

X 

X 

1 1 .  This  is  done  but  problems  come  in  later  when  new  capabilities  are  proposed, 
then  integration  problems  come  up. 
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When  asked  “What  is  important?”  when  presented  with  all  of  the  eards  it  was  tied  for 
tenth  with  six  out  of  the  ten  interviewees  that  responded  choosing  this  card.  When 
limited  to  only  their  top  three  choices  it  was  tied  for  third  with  only  three  interviewees 
selecting  this  card.  When  the  interviewees  were  asked  to  order  the  cards  by  when  they 
believed  the  activities  should  occur  it  was  third  out  of  sixteen. 

After  analyzing  the  comments  and  responses  to  the  questions  asked  the 
research  team  can  conclude  that  the  maturity  elements  in  the  framework  represented 
by  this  card  can  add  value  to  the  decision  making  process.  This  ranked  highly  for 
importance  to  the  interviewees  and  they  indicated  that  it  should  be  started  fairly  early 
during  the  acquisition  process. 

Exercise  Card  “L”  Results. 

A  popular  response  to  card  “L”  was  that  developers  do  not  currently  do  a  good 
job  at  the  activities  mentioned  in  this  card,  see  Table  17.  Some  respondents  said  the 
matrix  matching  described  in  the  card  would  be  a  good  communication  tool  for 
relating  requirements  to  capabilities.  However,  they  also  said  creating  the  matrix  and 
documenting  the  information  may  be  difficult  due  to  lack  of  details  early  on.  The  use 
of  architecture  products  to  capture  these  relationships  was  mentioned,  but  with 
complex  systems  these  products  can  get  overly  complicated  quickly  and  the  product 
becomes  the  focus  instead  of  the  information  it  contains. 
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Table  17  -  Qualitative  Comments  for  Exercise  Card  “L” 


A  matrix  matching  required  system  capabilities  to  the  operational  activities  that 

must  be  performed  to  achieve  them. 

Current 

Important 

Top  three 

Comments  by  interviewee  #  X  on  Maturity  Element  L 

X 

X 

X 

X 

3.  Not  a  very  good  job. 

X 

X 

4.  This  is  the  concept. 

X 

X 

X 

X 

X 

6.  This  is  done,  but  not  done  well.  We  need  to  trace  our  requirements. 

7.  Not  enough  technical  detail  this  early  on  to  develop  these  architecture  products. 

X 

X 

X 

8.  This  is  a  very  good  way  to  communicate.  We  don’t  do  it  much  currently,  but 
it’s  extremely  valuable. 

X 

X 

9.  This  needs  to  be  done  but  it  is  not  documented  well  currently  other  than  the  top 
level  requirements. 

X 

10.  This  may  get  complex  very  fast.  We  make  a  list  decently,  the  matrix  idea 
might  be  hard  to  really  make  understandable.  Danger  is  you  can  become  more 
obsessed  with  the  matrix  than  its  purpose. 

X 

11.  Post  AoA  is  where  the  requirements  traceability  happens. 

When  asked  “What  is  important?”  when  presented  with  all  of  the  cards  it  was  tied  for 
fourth  with  eight  out  of  the  ten  interviewees  that  responded  choosing  this  card.  When 
limited  to  only  their  top  three  choices  it  was  tied  for  seventh  with  only  two  out  of  the 
ten  interviewees  that  responded  choosing  this  card.  When  the  interviewees  were  asked 
to  order  the  cards  by  when  they  believed  the  activities  should  occur  it  was  fourth  out 
of  sixteen. 

Even  though  this  card  did  not  rank  very  high  in  importance  with  the 
interviewees,  their  comments  indicated  that  this  type  of  information  can  be  very 
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valuable.  The  respondents  described  the  activities  associated  with  card  “L”  as  being  a 
valuable  tool  for  decision  makers;  however,  they  also  indicated  that  these  activities  can 
be  a  very  time  and  resource  intensive  maturity  element  to  produce.  Regardless,  the 
researchers  have  determined  that  a  proper  requirements  traceability  matrix  does 
provide  value  to  the  decision  maker  when  choosing  between  concepts. 

Exercise  Card  “M”  Results. 

The  responses  indicated  that  developers  perform  satisfactorily  at  the  activities 
associated  with  card  “M”,  risk  management,  especially  once  the  system  has  been 
defined,  see  Table  18.  Respondents  mentioned  that  often  where  developers  get  into 
trouble  is  when  too  much  focus  is  placed  on  external  risks  to  the  system.  More  focus 
on  the  internal  risks  of  a  program  are  needed  since  when  risks  are  identified  the  focus 
is  external  versus  internal.  Additionally,  respondents  specifically  indicated  that 
assessing  cost  risks  are  currently  an  area  that  is  lacking. 
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Table  18  -  Qualitative  Comments  for  Exercise  Card  “M” 


A  risk  assessment  matrix  of  the  risky  elements  of  system  development  and 
integration  as  well  as  mitigation  strategies. 

Current 

Important 

Top  three 

Comments  by  interviewee  #  X  on  Maturity  Element  M 

X 

X 

X 

X 

3.  Very  good. 

X 

X 

X 

4.  The  desired  system. 

X 

X 

X 

X 

6.  This  is  done  well;  we  establish  this  early  and  often. 

X 

X 

X 

7.  What  is  the  development  risk?  This  gives  perspective.  Risks  identified  are 
focused  on  external  factors  instead  of  the  internal  risks  (there  needs  to  be  more 
introspection.)  We  don’t  look  at  what  we  can’t  do,  but  what  others  can’t  do. 

X 

X 

X 

8.  We  are  heavy  on  risk  and  we  do  a  good  job  at  it,  however,  we  do  not  do  cost 
risk  very  well. 

X 

9.  We  have  a  good  handle  on  it  but  only  after  the  system  has  been  defined. 

X 

10.  Good  idea  to  use  and  we  need  to  start  it  earlier. 

X 

X 

11.  We  do  risk,  but  the  AoA  doesn’t  address  mitigation  strategies  and  the  quality 
of  the  risk  assessments  vary  in  their  usefulness  (very  dependent  on  the  team  doing 
the  assessment). 

When  asked  “What  is  important?”  when  presented  with  all  of  the  cards  it  was  tied  for 
fourth  with  eight  out  of  the  ten  interviewees  that  responded  choosing  this  card.  When 
limited  to  only  their  top  three  choices  it  was  tied  for  third  with  only  three  out  of  the  ten 
interviewees  that  responded  selecting  this  card.  When  the  interviewees  were  asked  to 
order  the  cards  by  when  they  believed  the  activities  should  occur  it  tied  for  eighth  out 
of  sixteen.  The  activities  described  in  this  card  have  a  relatively  high  importance  to 
the  respondents  and  does  provide  value  for  decision  makers. 
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Exercise  Card  “N”  Results. 


The  comments  related  to  card  “N”  centered  on  its  importance  though  it  is  often 
delayed  till  later  in  development,  see  Table  19.  The  respondents  mentioned  that  not  a 
lot  of  work  is  put  into  this  upfront  and  is  usually  not  addressed  until  MS-B.  Several 
respondents  commented  that  these  activities  should  be  started  earlier  and  should  at  a 
minimum  contain  an  initial  measure  of  “goodness”  that  the  user  desires.  The 
interviewees  mentioned  that  there  has  to  be  some  MOEs  created  early  on  so  as  to 
effectively  screen  out  concepts  that  will  not  meet  the  user’s  need. 


Table  19  -  Qualitative  Comments  for  Exercise  Card  “N” 


Quantifiable  measures  of  effectiveness  (MOE)  for  the  system  and  derived 
measures  of  performance  (MOP). 

Current 

Important 

Top  three 

Comments  by  interviewee  #  X  on  Maturity  Element  N 

X 

X 

1.  One  of  the  most  time  intensive  tasks.  Decision  makers  need  to  know  what  the 
measures  are.  This  is  done  well  with  qualitative  data  but  can  be  subjective. 

X 

X 

3.  Fairly  good 

X 

4.  This  happens  near  the  end. 

X 

X 

X 

X 

6.  The  quality  is  hit  and  miss. 

X 

X 

7.  What  is  your  measure  of  goodness?  Not  a  lot  is  done  upfront,  not  until  MSB. 

You  need  to  have  some  idea  of  the  MOE  early  on. 

X 

8.  This  needs  to  be  done  but  should  be  done  earlier  and  we  need  to  hold  the  users 
responsible  for  doing  the  tradeoffs. 

X 

9.  Usually  this  comes  into  play  during  the  analysis  of  alternatives  but  it  can  still  be 
used  earlier  in  the  trade  studies. 

X 

10.  We  have  a  hard  time  getting  to  the  MOPs  from  the  MOEs  and  there  is  no  test 
plan  early  on.  Test  representatives  are  at  the  development  planning  groups  but 
they  often  think  it  is  a  waste  of  time  so  early  on  in  the  process. 

X 

X 

1 1.  Is  done  and  takes  a  lot  of  work  to  draft  MOEs  and  MOSs  but  it  is  critical.  The 
quality  is  variable  and  done  early  in  the  AoA  process. 
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When  asked  “What  is  most  important?”  when  presented  with  all  of  the  eards  it  was 
tied  for  thirteenth  with  five  out  of  the  ten  interviewees  that  responded  choosing  this 
card.  When  limited  to  only  their  top  three  choices  it  was  tied  for  ninth  with  only  one 
out  of  the  ten  interviewees  that  responded  choosing  this  card.  When  the  interviewees 
were  asked  to  order  the  cards  by  when  they  believed  the  activities  should  occur  it  was 
fourteenth  out  of  sixteen.  The  order  provided  by  the  respondents  is  inconsistent  with 
their  comments  and  could  possibly  be  due  to  what  they  currently  see  happening  in 
acquisitions  versus  what  they  believe  should  be  happening.  Further  research  with  a 
stronger  focus  on  studying  the  recommended  order  could  provide  additional  insight 
and  reduce  inconsistencies  with  respondents’  comments. 

Some  respondents  discussed  how  this  early  in  the  acquisition  process  decision 
makers  should  have  some  measure  of  “goodness”  that  they  can  judge  the  proposed 
concepts  against.  Even  if  the  MOEs  are  not  defined  early  on  decision  makers  can  still 
make  comparisons  between  concepts  and  their  performance.  Respondents  also 
mentioned  that  at  this  early  stage  (i.e.  pre-MS-A)  the  value  of  top  level  MOEs  can  be 
established,  but  the  MOPs  can  be  addressed  later  in  the  acquisition  process. 

Exercise  Card  “O”  Results, 

Card  “O”  was  not  a  part  of  the  Hughes’  Eramework  but  was  added  in  the 
interview  to  get  opinions  on  its  value  early  on  in  the  development  process  and  insight 
into  its  exclusion  from  the  framework.  Overall  the  responses  indicate  that  a  test  plan 
is  typically  not  started  this  early  in  development,  see  Table  20.  While  considerations 
should  be  made  that  the  MOEs  and  MOPs  should  be  able  to  be  tested,  initial  test  plans 
are  premature  since  the  concept  solution  has  not  even  been  chosen  yet.  Initial 
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involvement  of  testers  may  be  useful  to  ensure  that  the  measures  can  be  tested  but 
beyond  that  it  is  questionable. 


Table  20  -  Qualitative  Comments  for  Exercise  Card  “O” 


Initial  test  plan  for  how  to  evaluate  the  system  against  its  MOPs  and  EOEs. 

Current 

Important 

Top  three 

Comments  by  interviewee  #  X  on  Maturity  Element  0 

X 

X 

2.  More  emphasis  should  be  placed  on  this. 

X 

X 

X 

4.  This  happens  near  the  end. 

X 

X 

X 

X 

6.  This  is  not  done  early  and  is  the  lowest  priority. 

X 

7.  Not  a  lot  is  done  upfront,  not  until  MSB.  Definitely  don’t  need  a  lot  of  details 
in  measures  this  early  on;  testing  is  just  getting  off  the  ground  at  MSA. 

X 

8.  Involve  the  testers  earlier,  we  are  improving  at  this. 

X 

9.  This  is  not  needed  early  on. 

X 

10.  We  have  a  hard  time  getting  to  the  MOPs  from  the  MOEs  and  there  is  no  test 
plan  early  on.  Test  representatives  are  at  the  development  planning  groups  but 
they  often  think  it  is  a  waste  of  time  so  early  on  in  the  process. 

X 

11.  Test  people  have  started  an  early  involvement  effort,  but  their  time  is  limited. 

When  asked  “What  is  important?”  when  presented  with  all  of  the  cards  it  was  fifteenth 
with  four  out  of  the  ten  interviewees  that  responded  choosing  this  card.  When  limited 
to  only  their  top  three  choices  it  was  tied  for  ninth  with  only  one  out  of  the  ten 
interviewees  that  responded  choosing  this  card.  When  the  interviewees  were  asked  to 
order  the  cards  by  when  they  believed  the  activities  should  occur  it  was  fifteenth  out  of 
sixteen.  Although  a  test  plan  is  required  later  during  development,  the  majority  of 
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respondents  indicated  that  starting  it  too  early  would  be  premature  and  not  necessary. 
Similar  to  card  “J,”  these  results  support  the  maturity  elements  included  in  the 
framework. 

Exercise  Card  “P”  Results. 

The  tasks  and  activities  associated  with  card  “P”  was  chosen  as  one  of  the  most 
important  elements  that  should  be  considered  when  assessing  a  concepts  maturity,  see 
Table  21.  Nearly  all  respondents  said  that  they  must  understand  the  problem  first 
before  any  other  activities  are  performed.  However,  respondents  indicated  that  the 
quality  of  effort  currently  accomplished  with  the  activities  related  to  this  card  varies 
greatly.  The  main  reason  given  for  this  variation  is  the  experience  of  the  people 
charged  with  developing  these  products.  High  personnel  turnover  and  team  breadth  of 
expertise  problems  were  identified  as  possible  causes.  More  user  involvement  in  these 
activities  and  clearer  guidance  are  possible  solutions  that  were  recommended. 


90 


Table  21  -  Qualitative  Comments  for  Exercise  Card  “P” 


Concept  of  operations  (CONOPS)  describing  the  problem,  the  desired  effects, 
assumptions,  critical  capabilities,  enabling  capabilities,  sequenced  actions  and  the 

end  state. 

Current 

Important 

Top  three 

Comments  by  interviewee  #  X  on  Maturity  Element  P 

X 

X 

X 

1.  One  of  the  more  time  intensive  tasks.  How  you  will  use  the  system  (know  what 
the  critical  capabilities  are  and  how  they  are  implemented). 

X 

X 

X 

X 

X 

X 

X 

X 

X 

4.  Describes  the  problem  (ensures  everyone  is  on  the  same  page)  this  is  not  done 
with  consistent  quality  (we  are  more  solution  than  capability  driven). 

X 

X 

X 

5.  We  need  to  understand  the  problem  and  the  desired  end  state. 

X 

X 

6.  Because  of  personnel  turnover  it  is  not  done  as  well  as  it  has  been  done  in  the 
past. 

X 

X 

7.  This  describes  the  problem. 

X 

X 

X 

8.  The  acquisition  community  is  not  as  involved  with  the  development  of 

CONOPS  as  they  should  be.  Currently  we  are  stove-piped  into  solving  the  one 
capability  gap.  We  rarely  back  up  the  capabilities  needed  in  a  CONOPS. 

Capability  Based  Planning  is  addressing  this  but  it  is  currently  not  done  well. 
Development  Planning  does  this  well  but  the  Program  Managers  are  not  involved. 

X 

X 

X 

9.  Getting  the  CONOPS  and  its  context  are  critical.  It  is  very  hard  working  with 
the  MAJCOMs  on  requirements  because  they  don’t  have  the  people  committed  to 
changing  strategies,  and  they  are  too  focused  on  today’s  “Fires.” 

X 

10.  Is  done  better  than  most  and  the  users  need  to  be  involved  early  to  help. 

X 

X 

11.  Usually  there  is  an  employment  concept  working  group  for  replacement 
programs.  The  user  typically  wants  to  do  what  they  do  now  just  do  it  cheaper, 
better  and  faster.  For  new  problems  the  CBA  process  is  used  along  with  the  ICD 
to  capture  requirements. 

When  asked  “What  is  important?”  when  presented  with  all  of  the  cards  it  was  first 
with  ten  out  of  the  ten  interviewees  that  responded  choosing  this  card.  When  limited 
to  only  their  top  three  choices  it  was  first  with  eight  out  of  the  ten  interviewees  that 
responded  choosing  this  card.  When  the  interviewees  were  asked  to  order  the  cards  by 
when  they  believed  the  activities  should  occur  it  was  first  out  of  sixteen.  After 
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analyzing  the  comments  and  responses  to  the  questions  asked,  the  research  team  can 
conclude  that  the  maturity  elements  in  the  framework  represented  by  this  card  does 
add  value  to  the  decision  making  process. 

Additional  Comments. 

Table  22,  below,  lists  qualitative  comments  given  by  each  respondent  not 
directly  related  to  any  of  the  cards,  but  the  researchers  recorded  them  for  the  additional 
context  they  yield  surrounding  this  study.  A  common  pattern  observed  from  the 
interviewees’  comments  was  that  all  of  the  maturity  elements  in  the  Hughes’ 
Framework  can  provide  some  utility  to  decision  makers  if  they  are  done  properly. 


Table  22  -  Additional  Qualitative  Comments 
Other  comments  by  interviewee  #  X 

1 .  All  of  the  maturity  elements  provide  some  value  if  they  are  done  correctly. 

2.  Usually  you  see  problems  in  the  earliest  parts  of  the  process.  High  turnover  of  leadership  is  a 
problem  and  responsibilities  are  assigned  as  additional  duties  so  it  is  not  a  primary  concern.  Defining 
the  problem  can  take  months  if  given  bad  guidance.  Decision  makers  don’t  care  about  the  details;  they 
just  want  to  know  that  the  details  have  been  worked  out.  Far  too  often  too  much  effort  is  put  into  one 
maturity  element  at  the  expense  of  others  and  they  spend  too  much  time  on  areas  they  already  know 
about. 

3.  The  most  time  consuming  part  of  programs  was  milestone/program  reviews.  Every  review  usually 
wanted  its  own  format  which  adds  more  time.  All  maturity  elements  provide  some  value  but  to 
different  degrees. 

4.  Educate  the  people  doing  the  work  “more  training  in  early  systems  engineering.”  If  systems 
engineering  is  done  correctly  the  program  reviews  should  be  a  lot  easier  to  prepare  for.  Everything 
needs  to  be  documented  better.  Patience  is  important;  leaders  need  to  understand  it’s  going  to  take 
awhile. 

8.  Decision  makers  want  to  know  the  detailed  analysis  has  been  done,  but  does  not  need  to  know  the 
details.  It  is  a  new  idea  prior  to  MDD  to  be  thinking  about  all  of  these  maturity  elements. 

9.  All  the  elements  can  be  useful,  but  we  will  need  more  money  and  time  early  on  to  do  all  these  things 
well.  Right  now  there  isn’t  enough  money  budgeted. 

10.  I’d  like  to  do  all  of  these,  but  they  all  cost  money;  need  more  investment  up  front  to  accomplish. 
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Another  common  statement  made  was  “we”,  the  DoD  acquisition  community,  need  to 
educate  our  acquisition  workforce  better,  not  just  individuals,  but  how  to  work  as  an 
acquisition  team  towards  a  common  goal.  Furthermore,  if  the  DoD  wants  better 
systems  that  perform,  cost  and  are  delivered  when  promised;  then  they  need  to  provide 
the  resources  up  front  to  do  the  systems  engineering  work  necessary  to  make  sure  we 
are  defining  the  right  problem,  pursuing  the  right  solutions,  and  developing  the  right 
capabilities  the  users  need. 

Comments  on  Missing  or  Desired  Information. 

Table  23,  below,  captures  the  comments  the  interviewees  had  about  what  they 
believed  was  missing  or  should  be  added  to  the  framework  to  help  decision  makers 
make  better  choices.  Some  of  the  comments  came  from  a  misinterpretation  of  what 
the  cards  represented  with  respect  to  the  data  that  was  supposed  to  be  captured  by  the 
activities  listed  on  the  cards.  A  Concept  of  Employment  is  addressed  in  the 
Opportunity  Identification  phase  of  the  framework  represented  with  card  “P”.  An 
identification  of  stakeholders  occurs  in  this  phase  also  and  is  represented  by  card  “D”. 
“What  do  we  need”  needs  more  definition  is  covered  in  this  phase  also  with  cards  “P” 
and  “C”.  The  other  comments  in  Table  23  helped  highlight  general  themes  and  areas 
of  concern  used  in  the  following  conclusions  and  recommendations  chapters. 
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Table  23  -  Qualitative  Comments  for  Missing  Elements 


Comments  on  what  is  missing  by  interviewee  #  X 

1.  Add  “Concept  of  employment”  to  the  features  of  the  CONOPS 

2.  Identification  of  stakeholders 

3.  Team  development 

5.  “What  do  we  need”  needs  more  definition. 

7.  What  is  the  resource  commitment  to  the  problem?  What  is  the  acquisition  strategy?  Supportability  is 
ignored  early  on.  Configuration  management  plans  (who  owns  the  data?) 

8.  Identification  of  specialty  skill  sets  needed  to  solve  the  problem. 

10.  We  use  to  have  an  analytic  group  of  people  that  forecast  what  we  will  need  in  the  future,  where  did 
they  go? 


Quantitative  Results  to  the  Exercise  Cards 

The  following  three  histograms  (Figures  3,  4,  and  5)  capture  the  interviewees’ 
responses  to  the  questions  of  what  cards  are  done  currently,  what  are  important  to 
adding  value,  and  their  top  three  most  important  activities,  respectively. 


12 
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Figure  3  -  What  is  done  currently? 
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This  histogram  shows  the  aggregation  of  responses  when  asked  if  the  activities  and 
data  that  the  cards  represent  presently  occur  during  early  development,  see  Figure  3. 
For  the  most  part  the  respondents  said  the  majority  of  the  activities  are  performed  to 
one  extent  or  another.  This  histogram  does  not  intend  to  capture  the  level  of  quality  or 
difficulties  in  accomplishing  the  activities  associated  with  each  card,  please  refer  to  the 
previous  sections  on  each  card  for  further  discussion. 


Figure  4  -  What  is  important? 


Figure  4  shows  the  total  responses  concerning  the  important  activities  that  should  be 
performed  during  early  development.  These  activities  ultimately  support  and  inform 
the  choices  of  the  decision  maker.  One  interesting  observation  is  that  of  all  of  the 
maturity  elements  proposed  by  Hughes,  the  two  that  were  added  by  the  research  team 
to  see  if  there  were  any  elements  that  may  be  missing  from  the  framework,  “O”  and 
“J”,  ranked  the  lowest  in  their  importance.  Again,  this  figure  supports  that  the 
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framework  includes  maturity  elements  that  are  recommended  for  use  during  early 


development  and  excludes  activities  that  can  be  addressed  later. 


PCAI  KMELBFHNODGJ 


Figure  5  -  Top  three  choices 

Figure  5  shows  the  totals  when  respondents  were  limited  to  selecting  only  their  top 
three  most  important  activities.  According  to  the  respondents  these  activities  are  the 
most  important  sources  of  information  that  decision  makers  need.  This  histogram 
indicates  a  large  preference  towards  card  “P”  compared  to  the  others  with  a  fair  desire 
for  cards  “C”  through  “M”.  This  histogram  is  only  meant  to  supplement  Figure  4, 
“What  is  Important”,  by  offering  insight  into  where  respondents  think  maturity 
elements  should  be  prioritized. 

Responses  to  Ordering  of  the  Maturity  Elements 

Another  question  asked  of  the  interviewees  was  what  order  if  any  would  they 
assign  to  when  the  information  contained  in  the  cards  should  be  performed  in  order  to 
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make  a  well-informed  decision  on  a  concept’s  maturity.  The  interviewees  had  a  wide 
range  of  ordering  with  some  activities  occurring  in  parallel.  There  was  as  many  as 
eight  groups  or  steps  in  the  process  for  some  respondents  and  as  few  as  three  for  others 
with  activities  occurring  in  parallel.  For  comparison  the  research  chose  to  normalize 
the  order  for  each  interviewee.  The  range  given  by  the  interviewees  was  linearly 
normalized  from  zero  to  one.  Activities  that  were  identified  as  occurring  first  were 
valued  at  zero  and  the  activities  that  occurred  last  were  valued  at  one.  All  intermediate 
steps  were  normalized  linearly  between  zero  and  one.  Figure  6  shows  the  results  of 
this  ordered  process  from  first  to  last. 


Figure  6  -  Sequence  of  activities  (respondents) 


This  generalized  order  does  match  up  fairly  well  with  the  ordered  placement  of  the 
maturity  elements  of  the  Hughes’  Framework  with  a  few  exceptions  that  were 
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discussed  earlier  concerning  the  results  of  each  card.  Cards  “O”  and  “J”,  the  ones 
added  by  the  research  to  see  if  there  was  any  missing  maturity  elements  for  the 
framework,  were  again  found  at  the  end.  One  possible  flaw  in  this  chart  could  be  the 
respondents  applying  a  relationship  between  importance  and  order  they  should  occur, 
however,  the  research  team  believes  that  for  the  majority  of  the  respondents  the  order 
they  assigned  was  the  order  they  believed  it  should  occur  not  just  a  ranking  of 
importance. 

For  comparison,  the  research  team  also  attempted  to  create  a  normalized  order 
of  the  respective  maturity  elements  contained  within  the  Hughes’  Framework,  see 
Figure  7  below.  Instead  of  from  zero  to  one,  the  research  team  used  a  scale  of  one  to 
three,  where  each  interval  represents  one  of  the  three  gates  in  the  Hughes’  Framework. 
Additionally,  some  of  the  maturity  elements  are  referenced  in  multiple  stages.  For 
example,  card  “K”  is  indicated  in  stages  one  and  two  and  is  given  a  normalized  value 
of  1.5.  This  process  was  completed  for  each  card.  In  comparison  to  the  Figure  6, 
above.  Figure  7  lines  up  with  respect  to  the  interviewees  comments. 
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Figure  7  -  Sequence  of  activities  (Framework) 

The  research  team  recognizes  that  the  ordering  results  from  both  the 
perspective  of  the  respondents  and  the  framework  are  somewhat  awkward  and  are  a 
likely  area  for  criticism.  The  research  team  is  in  no  way  attempting  to  show  that  the 
general  correlation  between  the  two  ordering  histograms  is  a  reason  to  accept  the 
frameworks  order.  Instead,  the  researchers  chose  to  include  these  results  for 
discussion  and  to  propose  further  research  regarding  this  subject.  As  mentioned 
previously  the  intent  of  this  study  and  the  interviews  was  to  focus  on  the  content  of  the 
framework  over  the  stage-gate  approach.  In  conclusion,  the  research  team  proposes 
pursuing  research  focused  on  whether  or  not  the  stage-gate  approach  to  the  framework 
is  supported  by  practitioners  and  related  data. 
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Analysis  and  Results  from  Application  of  the  Framework 

The  majority  of  the  analysis  and  results  for  the  application  are  contained  within 
the  methodology  chapter  due  to  the  proprietary  nature  of  the  development  program 
discussed  and  the  need  to  speak  in  generalities  concerning  the  effort.  The  research 
team  reiterates  that  from  an  academic  perspective  applying  the  framework  was  not 
overly  complex  and  did  not  require  a  deep  background  of  the  concept  used.  Instead 
developing  the  products  involved  organizing  the  available  information  rather  than 
finding  or  creating  additional  data.  In  summary  the  application  of  the  framework  was 
successful  and  the  sponsoring  organization  was  pleased  with  the  results. 

Analysis  and  Results  of  Confirmatory  Sources 

We  can  look  to  current  Department  of  Defense  and  Air  Force  policy  and 
guidance  for  support  in  validating  the  Hughes’  Framework.  The  Joint  Capabilities 
Integration  and  Development  System  (JCIDS)  (CJCSM  3170.01D,  2009)  covers  the 
majority  of  the  acquisition  process  and  has  many  aspects  that  overlap  with  the  Hughes’ 
Framework.  The  Early  Systems  Engineering  (SE)  Guidebook  specifically  addresses 
the  timeframe  of  the  Hughes’  Eramework  from  the  initial  needs  identification  to 
Milestone  A  (SAE/AQ,  2009).  The  Concept  Characterization  and  Technology 
Description  (CCTD)  Guide  (SAE/AQ,  2010)  also  addresses  the  timeframe  up  to  the 
Milestone  A  decision  point,  but  it  begins  when  it  has  been  decided  that  the  solution  to 
the  need  will  be  a  materiel  solution.  The  Hughes’  Framework  preceded  the  official 
releases  of  the  CCTD  Guide  and  the  Early  SE  Guidebook.  However,  the  Early  SE 
Guide  was  available  in  draft  form  during  the  development  of  the  framework. 
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Maturity  elements  of  the  framework  are  present  in  current  policy  and  guidance 
to  one  extent  or  another.  We  can  look  at  each  phase  of  the  framework  and  compare 
the  maturity  elements  contained  in  each  phase  with  what  policy  and  guidance 
documentation  says  should  be  occurring.  This  comparison  will  lend  credibility  to  the 
maturity  elements  since  there  is  DoD  and  Air  Force  policy  that  support  their  use.  The 
Concept  Maturity  Framework  can  be  validated  as  a  useful  aid  to  the  decision  process  if 
the  maturity  elements  of  the  framework  are  supported  by  current  policy. 

In  the  Opportunity  Identification  phase  of  the  framework  the  questions  of 
“what  needs  to  be  done,  how  is  it  done  now,  and  who  is  involved?”  are  answered  with 
the  maturity  elements  in  that  phase.  The  question  of  what  needs  to  be  done  is 
answered  with  the  CONOPS  and  Operational  Activity  Gap  maturity  elements  in  the 
Hughes’  Framework.  In  the  JCIDS  process  these  maturity  elements  are  addressed  in 
the  Capabilities  Based  Assessment  (CBA);  the  CBA  is  the  analytic  basis  of  the  JCIDS 
process.  It  identifies  capability  needs  and  gaps  and  recommends  non-materiel  or 
materiel  approaches  to  address  gaps  (CJCSM  3170.01D,  2009).  The  Early  SE 
Guidebook  references  these  maturity  elements  as  part  of  the  CBA  activities  performed 
in  the  JCIDS  process  (SAE/AQ,  2009).  The  primary  purpose  of  a  CCTD  is  to  provide 
information  to  decision  makers.  The  CCTD  Guide  section  one  captures  the  first  step 
of  any  engineering  problem:  defining  the  problem.  Typically  the  information  will 
come  out  of  the  CBA  during  the  JCIDS  process  (SAE/AQ,  2010).  The  maturity 
elements  that  help  characterize  the  question  of  what  needs  to  be  done  are  shown  to  be 
supported  by  current  policy.  The  Early  SE  Guidebook  urges  developers  to 
characterize  the  need  and  capability  gap  through  a  CBA  during  the  JCIDS  process  and, 
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then  that  information  should  be  captured  in  a  CCTD  to  provide  decision  makers  the 
information  needed  to  make  their  choices. 

How  it  is  done  now  and  what  are  the  operational  risks?  These  questions  are 
addressed  in  the  framework  with  the  “Existing  Systems  Interfaces  and  Operational 
Risk”  maturity  elements.  The  JCIDS  process  addresses  this  question  in  Enclosure  A. 
CBA  Process  2.d  states  that  the  CBA  sponsor  must  perform  the  operational  assessment 
of  the  current  and  programmed  force  to  provide  the  required  capabilities,  identifying 
capability  gaps  and  potential  force  redundancies  for  each  scenario.  Einally,  the  CBA 
assesses  the  potential  operational  risk  associated  with  each  gap  (CJCSM  3170.01D, 
2009).  The  CCTD  Guide  addresses  these  questions  in  sections  3.3  and  7.1  where  they 
discuss  interfaces  and  operational  risk.  Section  3.3  describes  all  major  external  and 
internal  interfaces  necessary  for  a  successful  concept  solution.  It  identifies  those  that 
will  be  available  to  support  the  fielded  solution,  as  well  as  those  that  may  require 
additional  technology  development  and/or  AE  infrastructure  development.  Section  7.1 
of  the  CCDT  Guide  documents  the  risks  of  the  materiel  concept  satisfying  the 
capability  gap  in  the  operational  environment.  Eurthermore,  risk  is  addressed  with 
respect  to  completeness  of  the  definition  of  the  capability  need  statement  and 
associated  measures  (MOEs,  MOPs,  KPPs,  etc.)  (SAE/AQ,  2010).  The  Early  SE 
Guidebook  addresses  these  questions  in  sections  1.5.2  System  of  Systems  (SoS) 
Architectures,  3.4.1  Architecture  Characterization,  and  4.1  Risk.  Section  1.5.2 
discusses  the  need  for  SoS  architectures  to  encompass  the  internal  and  external 
relationships,  functions,  and  dependencies  of  all  the  constituent  systems.  Section  3.4.1 
asserts  that  once  all  the  concept  nodes  and  their  interfaces  have  been  analyzed. 
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investigation  of  the  system’s  potential  to  address  stated  needs/shortfalls  ean  now 
begin.  Simulating  the  concept  system  may  uncover  secondary  missions  for  the  new 
system,  expose  potential  vulnerabilities  to  enemy  countermeasures,  and  provide 
insight  into  satisfying  original  war- fighter  shortfalls.  Section  4.1  states  that  risk 
management  is  at  the  heart  of  technical  and  SE  planning.  During  this  phase  a  critical 
first  step  towards  affordable,  manageable,  and  executable  Technology  Development 
efforts  begin.  Risks  should  be  assessed  and  managed  as  described  in  the  DoD  Risk 
Management  Guide  and  the  AF  Risk  Management  Guide  (AFP AM  63-128)  (SAF/AQ, 
2009). 

Who  is  involved?  This  question  is  addressed  with  the  “Operational 
Constraints”  and  “Organizations  &  Services”  maturity  elements.  The  maturity 
element  addressing  the  operational  constraints  is  not  specifically  called  out  in  the 
Early  SE  Guidebook  or  JCIDS;  however,  the  CBA  process  includes  trade-space 
analysis  and  understanding  constraints  as  some  of  the  activities  that  should  be 
performed.  The  CCTD  Guide  does  address  operational  constraints  in  section  3.4 
where  it  states  that  an  understanding  of  user  needs,  constraints,  and  limitations  in  the 
operating  environment  assists  with  developing  MOEs  that  can  be  used  to  assess 
military  utility  of  a  concept,  including  operational  performance  (e.g.,  reliability, 
maintainability,  availability,  supportability,  sustainability,  deployability,  etc.) 
(SAF/AQ,  2010).  The  Organizations  &  Services  maturity  element  refers  to  the 
identification  of  any  organizations  involved  with  a  possible  solution  concept  and  the 
resources/services  that  will  be  expected  to  flow  between  them  for  success.  The 
CCTD  Guide  and  JCIDS  manual  address  in  depth  the  organizations  responsible  for 
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developing  the  solution  but  do  not  go  into  great  detail  about  the  organizational 
relationships  within  the  concept  solution  other  then  pointing  out  the  architectural 
products  such  as  OV-4s  and  OV-2s.  The  Early  SE  Guidebook  discusses  this  maturity 
element  in  section  1.5.1  titled  “SE  for  SoS”.  It  states  that  development  or  evolution  of 
SoS  capability  is  seldom  driven  solely  by  a  single  organization,  but  generally  involves 
multiple  Program  Executive  Officers  (PEO),  Program  Managers  (PM),  and  operational 
and  support  communities.  While  each  individual  stakeholder  group’s  objectives  and 
organizational  contexts  shape  its  expectations  with  respect  to  the  SoS,  any  one  group 
may  well  have  limited  knowledge  of  the  constraints  and  development  plans  for  the 
other  systems.  Planners  may  not  recognize  every  SoS  stakeholder,  or  may  not  realize 
that  a  particular  organization  or  group  needs  to  be  included  in  deliberations  (SAE/AQ, 
2009). 

As  concept  solutions  pass  through  the  first  gate  of  opportunity  identification 
the  previous  work  is  not  abandoned.  The  maturity  elements  are  updated  as  the  concept 
becomes  more  refined  and  the  solution  becomes  clearer.  These  maturity  elements  are 
the  foundation  for  which  the  solution  will  be  built  from. 

The  Concept  Screening  phase  prior  to  the  second  decision  gate  coincides  with 
the  Material  Development  Decision  (MDD).  In  the  previous  phase  of  the  framework, 
accurately  defining  the  need  was  the  major  activity.  In  the  Concept  Screening  phase 
characterizing  the  solution  for  a  specific  concept  is  the  focus.  The  Hughes’ 

Eramework  uses  DoDAE  architectural  viewpoints  to  assist  in  the  characterization.  The 
maturity  elements  identified  by  Hughes  that  support  this  solution  characterization  are 
Systems  Interfaces,  Eorm  &  Eunction,  Operational  Activity,  Organizational  Services, 
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and  Technology.  The  information  contained  within  these  maturity  elements  help 
develop  the  initial  identification  of  risk  and  the  initial  estimation  of  the  resources 
required  for  new  development  and/or  modification  of  existing  systems.  The  JCIDS 
manual  Enclosure  A,  paragraph  l.f,  states  that  the  CBA  should  use  the  existing  DOD 
Enterprise  Architecture  and  related  solution  architectures  as  means  of  assessing  the 
capability  gaps  and  proposed  approaches  to  mitigate  them  but  it  does  not  require  them 
to  be  completed  until  Milestone  B.  The  JCIDS  manual  discusses  cost  throughout  the 
process  but  addresses  the  maturity  elements  in  the  Hughes’  Eramework  in  Appendix  B 
to  Enclosure  B,  paragraph  4.C.3,  when  it  discusses  the  ownership  cost  as  a  Key  System 
Attribute  (KSA).  Identification  of  development  risks  is  not  specifically  addressed  in 
JCIDS  (CICSM  3170.01D,  2009).  The  Early  SE  Guidebook  specifically  addressed  the 
use  of  architecture  products  to  develop  and  describe  systems.  Section  3.6.2  states  that 
while  the  full  set  of  DoDAE  products  is  generally  unnecessary  for  purposes  of  early 
SE,  many  “views”  are  highly  relevant  when  maturing  concepts  for  the  purpose  of  an 
AoA.  A  number  of  these  products  identified  in  prior  steps  are  actually  used 
throughout  the  process  as  benchmarks  to  communicate  concept  maturity  and 
performance  as  the  concept(s)  gain  technical  fidelity  and  receive  approval  to  progress 
to  further  development  stages.  Initial  identification  of  development  risk  is  not 
discussed  in  the  Early  SE  Guidebook,  however,  risk  management  and  risk  mitigation 
strategies  are  discussed  in  section  4.1.  The  Early  SE  Guidebook  does  not  go  into 
detail  but  in  section  3.1  states  that  each  concept  developed  will  have  been  technically 
researched,  analyzed,  and  evaluated  against  a  validated  set  of  mission-based 
requirements,  and  costed  for  the  entire  life  cycle  (SAE/AQ,  2009). 
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As  concepts  pass  through  the  MDD,  which  corresponds  with  gate  two  of  the 
Hughes’  Framework,  it  has  been  decided  that  the  concept  solution  will  at  least  in  part 
be  a  materiel  solution.  The  last  phase  of  the  Hughes’  Framework  is  the  Concept 
Selection  phase  and  ends  at  Milestone  A,  which  corresponds  with  the  third  gate  in  the 
Hughes’  Framework.  The  first  maturity  element  in  this  phase  is  the  “Cost  /  Benefit 
Analysis  results.”  When  a  need  is  identified,  and  the  operational  risk  of  not  satisfying 
that  need  is  identified,  there  should  be  a  level  of  importance  associated  with  satisfying 
the  need.  The  “Cost  /  Benefit  Analysis  results”  is  another  aid  that  helps  pare  down  the 
solution  set  to  find  the  best  fit  at  the  right  price.  As  stated  in  the  Office  of  Aerospace 
Studies  Analyst’s  Handbook,  “Cost  is  never  a  measure  of  effectiveness”  but  what  can 
be  measured  is  the  cost  to  concept  effectiveness  comparison  (Office  of  Aerospace 
Studies,  2000).  Figure  8,  below,  shows  four  alternatives  and  the  cost  estimates  and 
effectiveness  estimates  for  each  alternative.  The  uncertainties  of  these  estimates  are 
represented  by  the  error  ellipses. 


(AF  Analyst’s  Handbook,  OAS,  2000) 
Figure  8  -  Is  the  increase  in  effectiveness  worth  the  increase  in  cost? 
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The  JCIDS  manual  does  not  specifically  reference  a  cost  benefit  analysis  but  it  does 
address  the  issue  in  Enclosure  B  when  talking  about  performance  attributes  and  key 
performance  parameters.  It  states  that  the  threshold  value  for  an  attribute  is  the 
minimum  acceptable  value  considered  achievable  within  the  available  cost,  schedule, 
and  technology  at  low-to-moderate  risk.  Performance  below  the  threshold  value 
neither  is  operationally  effective,  suitable  nor  provides  improvement  over  current 
capabilities.  The  objective  value  for  an  attribute  is  the  desired  operational  goal 
achievable  but  at  higher  risk  in  cost,  schedule,  and  technology.  Performance  above  the 
objective  does  not  justify  additional  expense  (CJCSM  3170.01D,  2009).  Both  the 
CCTD  Guide  and  Early  SE  Guidebook  discuss  cost  estimates  as  described  earlier  in 
this  section  but  they  do  not  reference  the  cost  benefit  analysis  results  other  than 
indicating  this  activity  occurs  during  the  Analysis  of  Alternatives  (AoA). 

The  “Technology  Development  Plans”  maturity  element  in  the  Hughes’ 
Eramework  is  represented  in  one  form  or  another  in  current  policy  and  guidance.  In 
the  Early  SE  Guidebook  this  maturity  element  is  referred  to  as  a  Technology 
Development  Strategy  (TDS).  In  section  four  it  states  that  the  TDS  is  the  foundation 
for  the  Acquisition  Strategy  and  eventually  the  Eife  Cycle  Management  Plan  (ECMP); 
it  contains  significant  detail  on  program  execution  during  the  Technology 
Development  (TD)  phase,  but  also  documents  early  planning  for  post  Milestone  B 
efforts.  Therefore,  it  must  include  all  activities  necessary  to  successfully  complete  the 
TD  phase  (SAE/AQ,  2009).  In  the  CCTD  Guide  this  maturity  element  is  referred  to  in 
section  6.2  as  the  Technology  Maturation  Approach.  It  states  that  the  technology 
maturation  approach  will  play  a  large  role  for  decision  makers  in  determining  where 
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the  concept  enters  the  acquisition  cycle,  as  it  describes  much  of  the  technical  work  that 
remains  to  mature  the  concept.  In  some  cases  the  path  forward  may  be  to  defer 
embarking  on  a  new  system  for  several  years  in  favor  of  investing  in  additional 
technology  efforts  (SAF/AQ,  2010).  The  JCIDS  manual  refers  to  a  TDS  as  well  but 
only  goes  into  as  much  detail  as  saying  that  the  TDS  is  dependent  on  the  ICD  and  that 
the  Capability  Development  Document  (CDD)  uses  the  TDS  as  an  input  (CJCSM 
3170.01D,  2009).  This  may  not  be  surprising  since  TDS  is  a  product  center  activity 
while  the  ICD  is  a  user  generated  document. 

The  “Risk  Management  Plans”  and  “Estimation  of  resources  required  for 
development,  operations,  and  sustainment  associated  with  the  proposed  solution” 
maturity  elements  are  continuations  of  the  work  started  in  the  Concept  Screening 
phase.  These  maturity  elements  are  updated  and  refined  in  the  Concept  Selection 
phase  as  the  candidate  solutions  are  narrowed  down  and  a  greater  definition  of  the 
solution  is  developed.  These  maturity  elements  are  supported  by  policy  and  guidance 
stated  earlier  in  this  section.  Though  the  identification  of  risks  was  not  specifically 
identified  in  some  of  the  policy,  risk  management  plans  were  identified  as  necessary 
elements  needed  for  system  development. 

Overall  there  is  an  abundance  of  support  for  the  maturity  elements  found  in  the 
Hughes’  Framework  for  concept  evaluation  and  selection.  The  Hughes’  Framework 
was  developed  from  current  Department  of  Defense  and  Air  Force  policy  and 
guidance.  Since  Hughes  developed  this  framework  nearly  all  of  this  policy  has  been 
revised  updated  or  approved  so  it  is  a  good  measure  that  the  DoD  and  Air  Force 
leadership  believe  that  these  elements  are  needed.  This  Concept  Maturity  Framework 


108 


provides  the  roadmap  for  obtaining  the  right  information  at  the  right  time  so  that 
decision  makers  have  the  necessary  information  to  evaluate  concept  solutions  and 
make  an  informed  selection. 
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V.  Conclusions 


While  seeking  to  answer  the  questions  described  in  the  research  objectives,  the 
research  team  discovered  much  more  along  the  way.  During  this  journey,  the  research 
team  gained  insight  by  talking  with  practitioners  of  the  DoD  acquisition  process, 
reviewing  policy  and  guidance,  and  by  gaining  real-world  experience  in  applying  the 
Hughes’  Framework.  These  insights  lead  to  patterns  and  themes  that  can  be  used  to 
gain  a  better  understanding  of  the  current  state  of  the  acquisition  process.  Along  with 
answering  the  research  questions,  the  team  will  share  some  themes  and  lessons-learned 
that  were  revealed.  These  themes  will  not  just  be  about  specific  aspects  of  the  concept 
maturity  framework  but  include  themes  that  encompass  a  larger  perspective. 
Validating  the  Hughes’  Framework 

Previous  research  proposed  a  “Concept  Maturity  Assessment  Framework” 
method  in  which  decision  makers  can  determine  the  maturity  of  concepts  as  they 
progress  through  the  acquisition  process.  This  study  uses  the  framework  to 
demonstrate  its  application  while  determining  the  utility  and  added-value  of  the 
framework  to  the  decision  maker.  As  part  of  the  framework  validation  effort  this 
study  sought  to  quantify  and  qualify  the  added-value  of  applying  the  framework 
assessment,  and  give  a  recommendation  for  or  against  future  use  during  the  acquisition 
process.  The  research  team  returned  to  the  research  questions  posed  earlier  in  this 
thesis  to  offer  recommendations  concerning  the  framework’s  validation. 

1.  Is  the  framework  a  valid  guide  that  can  be  used  to  assess  concept  maturity? 
Through  the  confirmatory  analysis  of  the  framework,  the  application  of  the 
framework  to  a  specific  program  in  development,  and  the  interviews  conducted  with 
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practitioners  throughout  the  acquisition  community,  the  research  team  can  infer  that 
the  Hughes’  Framework  is  a  valid  guide  that  can  be  used  to  assess  concept  maturity. 

2.  Can  the  framework  be  applied  in  real-time  during  concept  development? 
This  framework  is  designed  to  line  up  logically  with  the  early  decision  gates  in 

the  acquisition  process.  Some  of  the  maturity  elements  called  out  in  the  framework 
are  already  needed  at  these  decision  points  so  extra  effort  was  not  necessary.  Other 
maturity  elements  related  the  concept  capabilities,  functions,  form,  etc.  were  provided 
by  the  contracting  company  proposing  the  solution.  This  framework  can  be  applied  in 
real-time  during  concept  development. 

3.  What  is  the  added-value  to  the  decision  maker  in  applying  the  framework? 
The  added  value  to  the  decision  maker  is  a  reduction  of  uncertainty.  The 

decision  maker  may  not  care  or  want  to  know  about  all  of  the  detailed  analysis  that 
went  into  the  maturity  elements  in  the  framework,  but  if  he  or  she  is  certain  that  they 
have  been  considered  and  the  risks  are  properly  identified  then  they  have  sufficient 
information  to  make  a  well-informed  decision. 

4.  Does  the  framework  help  reduce  or  mitigate  the  risk  associated  with 
concept  maturity  for  the  decision  maker? 

The  answer  is  yes  for  the  same  reasons  mentioned  in  the  answer  to  question 

three.  The  framework  helps  reduce  uncertainty.  Uncertainty  contributes  to  risk. 

Reducing  the  uncertainty  reduces  the  unknown  risks. 

5.  Should  the  framework  be  recommended  for  use  during  the  acquisition 
process? 

Based  on  the  results  of  the  research,  the  framework  should  be  recommended 
for  use  during  the  acquisition  process.  The  framework  does  not  add  any  additional 
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requirements  to  the  process.  The  majority  of  the  information  called  out  in  the 
framework  is  required  at  some  point  in  the  acquisition  process.  What  the  framework 
does  is  help  the  user  systematically  order  their  thinking  about  the  elements  of  the 
system  earlier  in  the  process  and  suggests  a  series  of  actions  to  take.  This  effort  can 
help  identify  potential  problems  earlier  before  they  require  extensive  amounts  of 
resources  to  fix.  Furthermore,  by  using  the  information  maturity  elements  in  the 
framework,  risky  development  concepts  can  be  exposed  and  then  handled. 

The  research  team  finds  that  the  Hughes’  Framework  is  a  valid  tool  that  can  be 
used  by  acquisition  professionals  as  a  guide  for  assessing  the  maturity  of  concepts. 
This  framework  uses  maturity  elements  to  capture  information  about  concepts  in  real¬ 
time  as  the  concepts  are  matured.  These  maturity  elements  provide  valuable 
information  that  can  be  used  to  help  avoid  or  mitigate  the  development  of  risky 
concepts. 

Evaluating  Internal  &  External  Validity 

As  discussed  in  the  methodology  of  this  study,  the  research  team  would  test  for 
the  internal  validity  of  the  findings  from  the  interviews.  The  research  team  as  a  result 
of  the  interview  responses  accepted  the  primary  hypothesis  that  applying  the 
framework  adds  value  to  assessing  concept  maturity.  The  research  team  offers  an 
opinion  on  the  strength,  internal  validity,  of  this  conclusion  using  the  five  judgments 
as  discussed  by  Krathwohl  (1998),  see  Appendix  C.  Then  using  Krathwohl’s  five 
judgments  for  external  validity,  the  research  team  assesses  the  degree  to  which  this 
finding  can  be  generalized  past  the  controlled  interview  study,  see  Appendix  D.  After 
applying  the  five  judgments  for  both  sets,  the  research  team  claims  that  the  interview 
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approach  of  this  study  is  internally  valid.  More  importantly,  the  research  team 
concludes  that  the  findings  of  this  validation  effort  of  the  framework  can  be 
generalized  to  all  development  organizations  within  the  Air  Force  and  have  fair 
expectation  of  being  of  value  to  many  other  related  organizations  within  the  DoD  as 
well  as  non-government  organizations.  This  external  validity  of  the  framework 
promotes  further  use  for  both  practitioners  involved  in  early  development  and  decision 
makers  assessing  a  concept’s  maturity  level. 

Themes  and  Lessons-Learned 

As  mentioned  previously,  during  this  validation  effort  several  important 
themes  and  lessons -learned  were  discovered.  The  following  section  sheds  light  on  the 
context  behind  the  Hughes’  Framework  and  offers  some  best-practices  and  heuristics 
related  to  early  development. 

People  Make  the  Difference. 

A  common  refrain  from  respondents  was  “if  done  correctly;”  it  did  not  matter 
what  part  of  the  framework  was  being  discussed.  This  leads  to  a  discussion  about  the 
people  doing  the  work.  Does  the  Air  Force  and  the  DoD  have  the  right  people  doing 
the  early  development  work?  “We”  defined  as  the  acquisition  community  can  have  all 
the  tools  in  the  world  available  to  guide  us  through  the  acquisition  process,  but  if  we 
do  not  have  the  necessary  training  and  experience  to  use  these  tools  effectively  then 
we  are  hindering  the  process.  Nearly  all  respondents  said  “we  need  to  do  this  better” 
for  at  least  one  area  in  the  acquisition  process,  and  some  respondents  specifically  cited 
lack  of  experienced  and  motivated  personnel  doing  the  work  as  the  problem.  This  lack 
of  experience  was  not  concentrated  in  one  specific  area  of  acquisitions.  From  the 
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earliest  days  where  the  need  is  identified,  to  the  latter  stages  where  solution-specific 
cost  estimates  are  generated,  experience  and  motivation  of  the  teams  and  individuals 
performing  the  work  is  critical. 

Resources!  Resources!  Resources!. 

Another  common  statement  from  respondents  was  “if  there  is  time  and  if  there 
is  money  we  could  ...”  Systems  Engineering  (SE)  has  proven  its  worth  over  the  last 
few  decades,  and  the  overall  savings  that  are  generated  with  engineering  work  to  get 
the  design  right  the  first  time  has  been  demonstrated  frequently.  Honour  (2004) 
explores  the  value  of  SE  and  through  an  analysis  of  six  real-world  statistical  studies, 
he  concludes  applying  SE  activities  correctly  can  have  significant  added  value  to 
development  efforts.  Despite  this,  it  is  still  difficult  to  get  the  necessary  resources 
allocated  for  early  SE  work  to  make  sure  all  of  the  information  that  decision  makers 
need  gets  to  them.  This  information  should  be  thorough  and  accurate  but  due  to  the 
time  and  resource  constraints  that  are  placed  on  programs  this  may  not  always  be  the 
case.  This  kind  of  change  has  to  come  from  the  top  down,  but  the  acquisition 
community  needs  to  be  able  to  show  what  the  user  will  be  gaining  by  investing  more 
time  and  money  up  front. 

Are  we  using  the  resources  we  have  in  the  most  effective  way?  Another 
common  theme  was  that  we  spent  too  much  time  on  certain  activities  as  compared  to 
others  (i.e.  activities  A,  B  and  C  at  the  expense  of  X,  Y  and  Z).  One  thing  the 
researchers  did  in  this  study  was  attempt  to  assess  the  time  and  effort  spent  on  the 
various  maturity  elements  used  when  making  decisions  in  comparison  to  added  value 
of  those  maturity  elements  to  the  decision  maker.  Several  respondents  indicated  that 


114 


the  effort  required  to  at  least  begin  many  of  the  maturity  elements  during  early 
development  was  minimal  and  was  worth  the  cost.  However,  they  also  indicated  there 
needs  to  be  balance  and  that  spending  extensive  resources  on  detailed  analysis  too 
early  on  may  be  wasteful.  Honour  (2004)  echoes  this  study’s  respondents,  he  offers 
that  just  doing  SE  activities  for  their  own  sake  during  development  without  proper 
planning  and  quality  in  execution  can  be  ineffective.  Instead,  he  recommends  that 
great  gains  can  be  achieved  through  correctly  applying  quality  SE  efforts  (Honour, 
2004).  Finally,  Honour’s  (2004)  research  reaffirms  this  study’s  finding  that  effective 
use  of  resources  during  early  development  is  critical,  and  he  says  that  programs  should 
strive  to  spend  15-20%  of  development  resources  on  Systems  Engineering. 

I  Have  a  Great  Idea!. 

Another  theme  identified  from  the  interviews  was  the  idea  that  we,  the 
acquisition  community,  often  prescribe  a  system- specific  solution  very  early  on. 

There  was  an  impression  given  from  some  of  the  respondents  that  we  choose  a  system- 
specific  solution  as  a  crutch  to  ensure  resources  are  allocated  and  the  development 
program  continues.  A  solution  that  is  too  conceptual  and  does  not  have  a  specific 
system  tied  to  it  is  often  vulnerable  to  losing  interest  and  potentially  funding  from 
decision  makers.  However,  this  approach  puts  too  much  resource  emphasis  on 
describing  how  the  system  will  look  and  function  without  fully  understanding  the 
range  of  options  available  for  fulfilling  the  needs  of  the  user.  Prescribing  a  system- 
specific  solution  towards  the  start  of  a  development  effort  constrains  the  trade-space  to 
use  different  solutions. 
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We  Make  Paper!. 

“We  do  this  but  it  is  not  documented  well”  is  another  common  theme  from  the 
interview  respondents.  What  does  documenting  something  well  mean?  From  the 
responses  the  researchers  received,  far  too  often  the  engineers  or  people  doing  the 
work  thought  all  the  documentation  needed  was  the  final  report  or  briefing  so  that  they 
could  progress  through  one  review  gate  and  get  ready  for  the  next.  Within  the  DoD 
and  even  the  commercial  world  there  needs  to  be  a  push  towards  using  a  consistent 
tracking  of  changes  to  the  concept  as  it  develops  that  is  shareable  between 
organizations.  Model-Based  Systems  Engineering  (MBSE)  is  a  useful  method  to  help 
track  and  share  information  during  development  and  should  be  leveraged  more  by 
development  organizations.  Good  documentation  is  not  just  a  final  report  or  a 
briefing;  it  is  a  story  of  how  the  system,  concept  or  program  has  evolved  from  its 
earliest  iteration  to  its  current  form. 

What  level  of  detail  is  appropriate?  It  is  not  useful  to  just  document  the 
captured  data  from  a  maturity  element  if  the  right  level  of  detail  and  analysis  does  not 
go  with  it.  The  framework  and  many  respondents  during  the  interviews  proposed  that 
the  architecture  products  such  as  those  from  the  DoDAE  are  a  useful  way  to  capture 
this  information,  but  the  level  of  detail  prescribed  in  DoDAE  can  be  far  more  than 
what  is  necessary  this  early  in  the  acquisition  process.  A  balanced  approach  should  be 
taken  to  ensure  that  sufficient  detail  is  made  available  for  an  informed  decision.  Also 
this  information  needs  to  be  stored  and  built  upon  since  it  will  be  needed  in  the  latter 
stages  of  the  acquisition  process.  At  times  decision  makers  just  want  to  know  that  the 
analysis  was  done  and  not  the  details  of  the  analysis.  However,  this  is  not  to  preclude 
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US  from  doing  the  analysis  early  and  to  the  appropriate  level  of  detail  for  it  will  pay 
dividends  later  in  development. 

The  Glass  is  Half  Full. 

A  theme  brought  up  several  times  was  that  we  are  overly  optimistic  about  what 
can  be  done  during  development,  which  leads  to  problems  later.  Budget  and  schedule 
estimates  are  getting  better,  but  the  risks  that  affect  cost  and  schedule  should  be 
characterized  more.  Several  respondents  mentioned  that  a  lot  of  time  and  effort  is 
spent  characterizing  known  risks.  The  unexpected  risks  are  the  ones  that  can  cause 
havoc  because  existing  mitigations  strategies  may  not  be  adequate  enough  to  resolve 
them.  Better  documentation  and  analysis  early  on  would  help  balance  this  optimism 
and  uncover  problems  that  may  be  hidden. 
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VI.  Recommendations 


Improvements  to  the  Hughes’  Framework 

Based  on  the  themes,  lessons-learned,  and  validation  effort  in  general,  the 
research  team  offers  some  suggestions  for  improvement  as  part  of  the  formative 
evaluation  of  this  study.  The  framework  overall  adds  value,  but  there  are  some  areas 
where  further  clarifications  or  additional  detail  would  help: 

•  More  emphasis  on  needs  analysis  is  required.  The  problem  definition  and 
inputs  can  make  some  big  assumptions  when  attempting  to  describe  the 
need.  In  context  of  the  DoD,  The  framework  is  designed  to  take  an 
accepted  capability  gap  from  the  JCIDS  process  and  help  mature  a  concept 
to  fulfill  this  gap.  This  effort  is  described  in  the  framework  through 
activities  such  as  CONOPS  development  and  understanding  the  mission 
and  objectives.  Through  this  study  this  related  maturity  element  was 
lauded  as  the  most  important  and  often  overlooked  activity  during  early 
development.  The  framework  is  iterative,  but  the  research  team  concludes 
that  a  needs  analysis  and  a  problem  definition  are  the  launching  point  for 
all  other  maturity  elements  in  the  framework.  In  its  current  form  the 
framework  does  not  depict  a  notion  of  priority  or  emphasis  in  the  maturity 
elements.  Although  avoiding  assigning  priority  may  have  been  the  intent 
of  the  original  authors,  the  research  team  believes  the  framework  should  at 
least  capture  the  emphasis  required  on  needs  analysis. 

•  Cards  “J”  and  “O”  were  added  to  gauge  the  value  they  may  provide  in 
assessing  concept  maturity.  The  research  has  shown  that  this  information 
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represented  by  the  cards  did  not  add  any  significant  value  early  in  the 
development  process.  Although,  the  activities  related  to  these  cards  have 
value  and  are  required  later  on  in  development,  the  research  team  does  not 
recommend  adding  maturity  elements  “J”  and  “O”  to  the  framework. 

•  An  explanation  of  the  level  of  detail  required  for  each  maturity  element, 
based  on  best-practices  and  heuristics  gathered  from  practitioners.  The 
guidance  in  the  DoDAF  manual  is  not  sufficient  to  describe  how  to 
generate  the  maturity  elements  in  the  Hughes’  Framework.  The 
architecture  view  by  gate  (Table  2,  previously)  needs  more  than  just  saying 
“updated”.  A  description  of  the  possible  additional  information  that  should 
be  added  could  be  useful.  This  can  help  focus  resources  on  areas  that  are 
most  important  to  assessing  the  maturity  of  concepts,  and  prevent  further 
instances  spending  too  much  effort  one  activity  at  the  expense  of  another. 

Recommendations  for  Further  Research 

•  Propose  research  to  recommend  a  prioritization,  resource  allocation,  and 
ordering  of  maturity  elements 

As  discussed  previously,  there  is  lack  of  prioritization  concerning  the  maturity 
elements  in  the  Hughes’  Framework.  The  framework  breaks  up  the  maturity  elements 
by  gates  but  does  not  tell  the  practitioner  where  they  should  assign  the  greatest  effort 
or  resources.  This  study  lays  the  foundation  for  what  respondents  perceived  as  more 
important.  Further  research  could  explore  this  prioritization  based  on  different  types 
of  organizations. 
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Furthermore,  the  research  team  attempted  to  understand  if  the  respondents 
perceived  an  ordering  of  activities  based  on  schedule  and  inputs/outputs.  However, 
the  results  were  marginal  and  further  research  more  focused  on  this  ordering  could 
help  practitioners  understand  what  activities  should  come  before  others  or  if 
accomplishing  them  in  parallel  is  preferred.  Also,  this  research  should  focus  on  the 
inputs  and  outputs  of  the  activities  in  the  framework  and  how  they  affect  other 
activities.  Finally,  understanding  the  organizations  or  people  that  receive  and  deliver 
the  necessary  information  during  early  development  could  help  improve  the 
framework’s  utility. 

•  Propose  research  to  understand  dijferences  and  similarities  of  the 
framework’s  utility  to  the  commercial  sector 

This  study  was  focused  on  validating  framework’s  added  value  to  the  DoD. 

The  authors  of  the  framework  intended  it  to  be  of  value  to  commercial  development 
sector  as  well.  The  implications  from  this  study  have  shown  that  there  should  be  value 
in  applying  the  framework  to  developers  in  the  commercial  world,  but  additional 
research  would  support  this  claim.  A  similar  interview  approach  could  be  applied  to 
individuals  involved  in  early  development  employed  in  the  commercial  world.  The 
interview  could  be  tailored  to  better  understand  the  possible  differences  and 
similarities  of  using  the  framework  between  DoD  and  commercial  early  developments. 
This  research  could  also  lead  into  the  creation  of  a  concept  maturity  framework 
tailored  specifically  for  the  commercial  world 
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•  Propose  research  to  develop  an  explanation  of  the  level  of  detail  required 
for  each  maturity  element,  based  on  best-practices  and  heuristics  gathered 
from  practitioners 

The  guidance  in  the  DoDAF  manual  is  not  sufficient  to  describe  how  to 
generate  the  maturity  elements  in  the  Hughes’  Framework  or  to  what  level  of  detail 
they  should  be  prepared.  There  needs  to  be  some  type  of  “Framework  Handbook” 
similar  to  the  CCTD  Guide  and  Early  SE  Guidebook  that  includes  explanations  for 
applying  the  framework.  A  description  of  the  possible  additional  information  that 
should  be  added  could  be  useful.  Additionally,  further  research  could  explore  a 
method  to  integrate  the  framework  into  these  DoD  approved  guides.  Research  into 
this  area  may  help  practitioners  accomplish  the  activities  in  the  framework  to  the 
appropriate  level  of  detail  for  decision  makers  assessing  the  maturity  of  concepts. 
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Appendix  A:  Interview  Structure  &  Questions 

1 .  Explain  the  purpose  of  the  interview 

a.  To  understand  how  a  practitioner  perceives  a  concept  progresses  through  the  acquisition 
process  currently  or  recently  and  if  there  is  a  way  to  improve  this  process 

b.  Focus  on  concept  maturity  not  system  design  (i.e.  after  a  need  is  identified  and  pre-MSA) 

c.  Focus  on  how  the  practitioner  perceives  activities  as  adding  value  in  information  to  the 
decision  maker 

2.  Gather  contextual  and  scope  information 

a.  Position  and  Organization 

b.  Current  Responsibilities 

c.  Prior  projects/program  and  experience 

3.  Explain  Exercise  #1  (As-Is  Process) 

a.  Ask  interviewee  to  choose  cards  they  think  are  currently  accomplished  at  some  form  or 
level  during  the  early  stages  of  development  (i.e.  after  a  need  is  identified  and  pre-MSA) 

b.  No  “correct”  answer,  based  on  interviewee’s  perception 

c.  All,  some,  or  none  of  the  cards  can  be  used 

d.  Ask  interviewee  if  they  do  not  understand  the  meaning  of  any  cards  and  explain  briefly 

e.  As  they  pick  cards  ask  the  interviewee  to  comment  on  how  well  or  bad  each  card  is 
accomplished,  how  often,  how  difficult,  or  any  other  relevant  information 

4.  Begin  Exercise  #1 

a.  Record  all  responses  including  the  letter  designator  of  each  card  and  related  comments 

b.  Record  all  responses  unrelated  to  the  cards  separately 

5.  Explain  Exercise  #2  (To-Be  Process) 

a.  Ask  interviewee  to  arrange  the  cards  in  the  order  (if  there  is  an  order,  can  group  cards) 
they  think  is  most  efficient  and  effective  at  adding  value  as  information  to  a  decision 
maker  determining  if  a  concept  is  mature,  well  developed,  or  if  the  concept  needs  further 
work. 

b.  Remind  interviewee  to  only  include  cards  they  perceive  as  adding  value  and  that  coming 
up  with  an  order  is  optional  and  only  necessary  if  the  interviewee  believes  an  order  is 
desirable  (order  refers  to  the  schedule  of  activities,  not  a  ranking  of  priority) 

c.  Again  related  to  early  development  (i.e.  after  a  need  is  identified  and  pre-MSA) 

d.  No  “correct”  answer,  based  on  interviewee’s  perception 

e.  All,  some,  or  none  of  the  cards  can  be  used 

6.  Begin  Exercise  #2 

a.  Record  all  responses  including  the  letter  designator  of  each  card  and  related  comments 

b.  Record  the  order  or  grouping  of  cards  (if  applicable) 

c.  Record  all  responses  unrelated  to  the  cards  separately 

7.  Follow-on  Questions  for  Exercise  #2 

a.  Ask  interviewee  to  select  the  top  3  most  important  activities  to  adding  value  to  a  decision 
maker  being  able  to  assess  the  maturity  of  the  concept  and  why?  The  three  activities  that 
they  would  want  their  people  to  spend  the  most  time  developing  and  refining 

b.  Ask  interviewee  if  there  is  anything  not  in  the  cards  that  stands  out  that  they  would  want  to 
include  during  early  development  (i.e.  after  a  need  is  identified  and  pre-MSA) 


8.  Wrap-Up  and  Conclude  Interview  (ask  for  any  final  comments) 
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Appendix  B:  Interview  Exercise  Cards 


A 

B 

Rough  order  of  magnitude  initial  cost 
and  schedule  estimates. 

Descriptionof  interfaces  between 
system  elements  and  the  resources  that 
flow  between  them  as  well  as  the 
operational  activities  they  support. 

C 

D 

List  of  assumptions,  constraints  and 
enabling  capabilities  that  are  required 
for  the  system  to  operate  effectively. 

Organizational  relationships,  their  roles 
and  responsibilities,  and  how  the  flow 
of  resources  are  expected  to  occur. 

'X. 

E 

F 

Identification  of  the  system  as  a  whole, 
the  elements  that  make  up  the  system 
and  their  interconnections  internally 
and  external  to  the  systems. 

A  decomposition  of  all  operational 
activities  or  tasks,  and  the 
inputs/outputs  between  them. 

H 

G 

A  description  of  the  functions 
performed  by  systems  and  the 
interactions/resource  flows  required  to 
perform  that  function. 

A  mapping  of  desired  system 
capabilities  to  the  operational  activities 
that  must  be  performed  and  the  system 
functions  that  support  them. 

Figure  8.  Interview  Cards  A-H 
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1 

A  listing  of  maturing  technologies  that 
the  development  of  the  system  may  be 
dependent  upon. 

J 

A  listing  of  all  industry,  national  and 
international  standards  that  apply  to 
the  system. 

K 

A  listing  of  capabilities  needed  to  solve 
the  problem  decomposed  down  to 
lower  level  enabling  capabilities  and  the 
dependencies  between  them. 

L 

A  matrix  matching  required  system 
capabilities  to  the  operational  activities 
that  must  be  performed  to  achieve 
them. 

M 

N 

/ 

e 

\  risk  assessment  matrix  of  the  risky 
lements  of  system  development  and 
integration  as  well  as  a  mitigation 
strategies. 

Quantifiable  measures  of  effectiveness 
(MOE)  for  the  system  and  derived 
measures  of  performance  (MOP) 

0 

initial  test  plan  for  how  to  evaluate  the 
system  against  its  MOPs  and  MOEs 

p  :** 

Concept  of  operations  (CONOPS)  * 
describing  the  problem,  the  desired 
effects,  assumptions,  critical 
capabilities,  enabling  capabilities, 

f  f  1 

sequenced  actions  and  the  end  state 

OMTI - T<w* 

Figure  9.  Interview  Cards  I-P 
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Appendix  C:  Five  Judgments  for  Internal  Validity 
Explanation  Credibility 

The  first  judgment  for  internal  validity  is  explanation  credibility.  Krathwohl 
(1998)  claims  this  judgment  is,  “the  plausibility  of  a  proposed  relationship’s 
explanation  or  rationale,  built  as  it  is  on  previous  research  and  thought”  (p.l40). 
Krathwohl  explains  that  this  judgment  deals  with  the  credibility  of  the  question, 
hypothesis,  or  prediction  that  the  study  is  centered  on.  He  says  that  this  first  judgment 
is  critical  as  the  rest  of  the  study  depends  on  it.  The  research  team’s  hypothesis  that 
the  framework  adds  value  is  a  reasonable  causal  relationship.  The  cause,  applying  the 
framework  early  on  during  development,  can  lead  to  the  effect,  being  able  to  assess  the 
maturity  of  a  concept.  The  hypothesis  makes  no  claim  to  the  framework  ensuring  the 
concept  is  “good”  or  will  guarantee  a  successful  product.  The  hypothesis  only  claims 
that  the  framework  can  help  assess  if  a  decision  maker  has  enough  information  to 
properly  understand  a  given  concept.  Based  on  the  Krathwohl’s  reasoning  the 
research  team  claims  to  have  a  credible  explanation  or  hypothesis. 

Translation  Fidelity 

The  next  judgment  is  translation  fidelity,  “the  faithfulness  of  the  design  choices 
to  the  meaning  of  the  .  .  .  hypothesis  ...  as  defined  by  the  study’s  explanation  or 
rationale”  (Krathwohl,  1998,  p.l44).  This  judgment  relates  to  the  believability  of  the 
study’s  design  used  to  defend  the  hypothesis  (Krathwohl,  1998).  Krathwohl  offers 
that  evaluating  the  “six-links”  of  a  study’s  design  can  help  determine  translation 
fidelity. 
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Subjects,  the  Who  Link. 

The  first  link  is  the  subjects  or  the  who  of  the  study.  Krathwohl  (1998)  says 
that  the  subjects  should  be  examples  of  individuals  to  whom  the  hypothesis  would 
apply.  In  this  study,  the  subjects  are  the  practitioners  who  are  or  have  been  directly 
involved  in  work  related  to  early  development,  which  relates  directly  to  this  study’s 
hypothesis. 

Situation,  the  Where  Link. 

The  second  link  is  the  situation  or  the  where  of  the  study  (Krathwohl,  1998). 
Rather  than  a  physical  location  for  the  study,  the  researchers  discuss  the  situation  as 
the  subject’s  present  and  past  experience  with  activities  or  programs  related  to  early 
development.  The  researchers’  asked  the  subjects  to  reflect  on  these  experiences  and 
knowledge  to  base  their  responses  on. 

Treatment,  the  Why  Link. 

Next,  is  the  treatment  link,  which  reflects  the  why  of  the  study  (Krathwohl, 
1998).  Krathwohl  mentions  that  this  link  relates  to  the  cause  in  the  hypothesis,  which 
again  is  applying  the  framework.  The  “To-Be”  exercise  corresponds  to  this  treatment 
or  cause  in  the  sense  that  each  respondent  was  choosing  maturity  elements  in  the 
framework  as  adding  value. 

Observations  and  Measure,  the  What  Link. 

The  fourth  link  is  observations  and  measures  or  the  what  of  the  study  that 
Krathwohl  (1998)  claims  relates  to  the  effect  in  the  hypothesis.  The  effect  in  this 
study  is  adding  value  to  assess  concept  maturity.  The  measures  and  observations  then 
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are  the  aetual  eards  eaeh  respondent  seleeted  in  the  “To-Be”  exereise  as  they  map  to 
the  maturity  elements  in  the  framework. 

Basis  for  Sensing  Attributes  or  Changes,  the  How  Link. 

Krathwohl’s  (1998)  fifth  link  is  the  basis  for  sensing  attributes  or  ehanges  or 
the  how  of  the  study.  This  link  deals  with  how  the  study  is  conducted  and  the  idea  of 
determining  if  a  cause  implies  the  effect  (Krathwohl,  1998).  The  research  team  offers 
that  the  card  exercise  methodology  lets  each  respondent  select  the  maturity  elements 
they  desired,  which  were  then  used  to  judge  the  frameworks  overall  value.  This 
method  is  focused  on  the  content  within  the  framework  as  opposed  to  simply  asking 
each  respondent  if  they  would  recommend  the  framework.  The  respondents  have 
direct  experience  with  the  maturity  elements  and  not  the  framework.  Therefore,  it  is 
appropriate  to  ask  them  questions  related  to  the  elements  rather  than  the  framework, 
which  is  how  the  research  team  chose  to  structure  the  interview  study. 

Procedure,  the  When  Link. 

The  final  link  for  assessing  a  design’s  translation  fidelity  is  the  procedure  or 
when  of  the  study.  Krathwohl  (1998)  describes  this  link  as  the  mles  that  the  study  was 
conducted  under  as  they  demonstrate  a  standardized  and  consistent  process.  All 
individuals  interviewed  during  the  framework  study  were  asked  the  same  questions 
and  engaged  in  the  same  order  of  exercises  and  so  on.  The  only  deviation  was  the 
face-to-face  as  opposed  to  telephone  contact  method.  The  research  team  does  not  see 
this  deviation  as  a  degrading  the  procedures  of  the  interviews.  In  summary  the 
research  team  affirms  that  the  study  passes  the  second  judgment,  translation  fidelity, 
based  on  the  six  design  links  approach. 
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Demonstrated  Result 


The  third  judgment  as  discussed  by  Krathwohl  (1998)  is  a  demonstrated  result. 
Krathwohl  proposes  that,  “a  Demonstrated  result  appears  when  the  evidence  is 
authentic,  there  was  precedence  (or  concurrence)  of  cause,  and  an  effect  occurred  as 
was  expected  in  terms  of  the  relationship  described  by  the  hypothesis  .  .  (p.l47). 

From  this  proposal  he  gathers  that  a  demonstrated  result  has  strong  evidence  if  four 
attributes  can  be  shown. 

Authenticity  of  Evidence. 

The  first  attribute  is  the  authenticity  of  evidence  and  ensures  that  the  data  and 
evidence  gathered  during  the  study  is  believable,  unaltered,  and  from  the  stated  source 
(Krathwohl,  1998).  The  researchers  recorded  the  responses  of  each  respondent  and 
compared  notes  to  ensure  responses  were  accurate.  The  interviews  were  conducted  by 
the  research  team  rather  than  outsourced  to  an  outside  party  ensuring  that  the 
responses  came  directly  from  the  stated  individuals. 

Precedence  of  Cause. 

The  second  attribute  is  a  precedence  of  cause  and  Krathwohl  (1998)  offers  that 
this  attribute  is  nearly  impossible  to  prove,  but  can  be  reasonably  accepted.  This 
attribute  demands  that  the  cause  always  precedes  the  effect.  In  terms  of  the  study’s 
hypothesis,  the  research  team  finds  this  attribute  somewhat  arbitrary  since  violating 
the  attribute  would  require  that  the  effect,  the  framework  adds  value,  is  an  absolute. 
Since  value  is  a  matter  of  opinion  that  can  only  be  accepted  and  never  proven,  the 
research  team  ignored  this  attribute. 
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Presence  of  Effect. 


The  next  attribute  is  the  presence  of  effect,  which  Krathwohl  (1998)  describes 
as  being  able  to  show  that  the  desired  effect  actually  occurred  during  the  study  or 
experiment.  For  this  study,  the  effect,  the  framework  adds  value,  is  demonstrated 
through  the  conjoint  analysis  concept  of  aggregation  as  discussed  in  Marder  (1999). 
Essentially,  if  the  interview  responses  during  the  “To-Be”  exercise  supported 
accepting  the  maturity  elements  of  the  framework  as  adding  value,  then  one  could 
accept  the  framework  as  adding  value. 

Congruence  of  Explanation  and  Evidence. 

The  final  attribute  leading  towards  a  demonstrated  result  is  congruence  of 
explanation  and  evidence  (Krathwohl,  1998).  This  last  attribute  focuses  on  if  the 
cause  and  effect  are  reasonable  and  if  the  evidence  presented  logically  supports  the 
findings  of  the  study  (Krathwohl,  1998).  Krathwohl  discusses  that  if  the  conceptual 
leap  one  must  jump  to  reasonably  accept  the  explanation  given  is  small  then  the 
evidence  for  this  attribute  is  strong.  For  this  study,  one  could  reasonably  accept  that 
performing  activities  and  creating  products  of  information  concerning  a  particular 
concept  would  help  one  estimate  how  ready  or  “mature”  he/she  believes  this  concept 
to  be.  To  summarize,  the  research  team  finds  that  the  four  attributes  of  the 
demonstrated  result  judgment  should  be  accepted. 

Rival  Explanations  Eliminated 

Rival  explanations  eliminated  is  the  fourth  judgment  for  internal  validity  and 
Krathwohl  (1998)  claims,  “for  the  projected  explanation  or  rationale  to  be  accepted,  all 
reasonable  rival  or  alternative  explanations  of  the  data  must  be  eliminate”  (p.l48). 
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Krathwohl’s  definition  for  this  judgment  appears  unlikely  to  eliminate  every  possible 
alternative  explanation  for  the  data.  However,  he  offers  that  this  judgment  is  meant  to 
help  the  researcher  increase  the  strength  of  their  results,  rather  than  force  the 
researcher  to  explore  every  explanation  in  the  realm  of  possibility,  which  would  be  a 
task  that  could  take  an  eternity.  Instead,  the  research  team  looked  for  weaknesses  in 
the  structure  and  inferences  made  of  the  interviews  and  discusses  how  these  possible 
weaknesses  do  not  degrade  the  core  conclusions  of  this  study. 

The  first  area  of  concern  is  the  mapping  of  the  maturity  elements  from  the 
framework  to  the  exercise  cards.  This  mapping  is  fundamental  to  any  conclusions 
made  on  the  value  of  the  framework.  If  this  mapping  is  in  error  then  there  would  be 
no  connection  to  the  interview  responses  and  the  framework.  The  research  team, 
however,  as  discussed  in  the  methodology  explains  the  details  of  the  mapping  to 
ensure  all  content  within  the  framework  was  represented.  Another  weakness  is  that 
hypothesis  for  this  study  is  that  the  framework  adds  value  to  decision  makers  assessing 
concept  maturity.  Though  some  individuals  interviewed  had  experience  in  a  decision¬ 
making  role,  one  could  argue  that  the  study  was  conducted  solely  from  a  practitioner’s 
point  of  view.  One  could  further  argue  that  these  practitioners  cannot  accurately 
comment  on  what  is  of  value  to  a  decision  maker  in  early  development.  The  research 
team  proposes  that  these  practitioners  are  the  ideal  candidates  for  this  study  as  they 
have  first-hand  knowledge  with  what  information  their  bosses  (decision  makers)  want 
to  have  available  and  more  importantly  how  to  get,  create,  or  organize  it.  In  closing, 
not  every  alternative  explanation  for  the  findings  of  this  study  could  be  eliminated,  but 
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the  research  team  does  not  recognize  any  rival  explanations  or  weaknesses  that  destroy 
the  validity  of  this  study. 

Credible  Result 

The  final  judgment  discussed  by  Krathwohl  (1998)  is  a  credible  result,  and  is, 
“a  judgment  that  sums  up  the  four  earlier  judgments  and  asks  whether  in  terms  of 
external  prior  evidence  we  can  believe  the  result”  (p.  148).  The  research  team 
evaluated  the  study  using  the  previous  judgments  and  determined  that  each  passed 
with  fair  to  strong  evidence.  The  framework  was  developed  only  recently  and  this 
study  is  the  first  effort  to  assess  its  value  based  on  an  interview  approach.  Therefore, 
there  is  no  prior  evidence  or  a  similar  study  available  to  support  or  negate  the  findings. 
Regardless,  the  research  team  is  confident  that  the  findings  of  the  interview  study  are 
internally  valid. 
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Appendix  D:  Five  Judgments  for  External  Validity 
Explanation  Generality 

The  first  judgment  for  external  validity  according  to  Krathwohl  (1998)  is 
explanation  generality  and,  “is  a  judgment  of  the  plausibility  of  the  generality  that  is 
claimed,  implied,  or  inferred  for  the  relationship”  (p.l77).  Krathwohl  discusses  that 
this  judgment  is  the  most  important  to  being  able  to  generalize  any  findings  from  a 
study  as  the  hypothesis  must  be  logical  and  reasonable  to  apply  to  areas  outside  of  the 
controlled  study.  This  study’s  hypothesis  is  broad  enough  to  encompass  all  areas 
related  to  early  development.  During  the  interviews  the  focus  of  the  research  was  on 
practitioners  with  experience  in  early  development  mainly  involved  in  Air  Force 
organizations.  However,  the  hypothesis  and  intent  of  the  research  would  not  preclude 
external  organizations  such  as  other  military  branches  of  service  (i.e.  Army  and  Navy), 
DoD  development  agencies,  and  even  commercial  industry.  Using  Krathwohl’s 
rationale,  the  research  team  finds  this  study  to  satisfy  explanation  generality. 
Translation  Generality 

The  second  judgment  is  translation  generality  and,  “is  a  judgment  of  the  extent 
to  which  the  generality  claimed,  implied,  or  inferred  in  the  study  is  represented  in  the 
operational  choices  of  its  design”  (Krathwohl,  1998,  p.l80).  Krathwohl  offers  that  this 
judgment  similar  to  translation  fidelity  for  internal  validity  focuses  on  how  the 
structure  or  design  of  the  study  is  appropriate  to  generalize.  Also,  similar  to 
translation  fidelity  he  cites  that  six  facets  to  the  design  can  help  explore  a  study’s 
translation  generality. 
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Subjects  and  Situations. 

Krathwohl  (1998)  combines  these  facets  for  external  validity  and  claims  that 
strong  external  validity  the  subjects  and  situation  used  during  the  study  should  not  be 
narrow  and  restrictive.  The  subjects,  the  practitioners  involved  in  early  development, 
were  primarily  from  Air  Force;  but  they  were  from  a  variety  of  organizations  and  prior 
experiences.  Many  of  these  individuals  worked  with  or  had  worked  with  other 
military  branches  and  DoD  agencies  and  understood  how  the  joint  development 
operates  in  the  DoD.  In  terms  of  the  situation  facet,  many  of  the  respondents  had 
direct  experience  with  joint  early  development  projects  and  programs  to  base  their 
responses  promoting  the  external  validity  of  this  study’s  findings. 

Treatment. 

Whereas  standardization  and  a  consistent  experimental  approach  promotes 
strong  internal  validity,  too  much  can  restrict  the  generality  of  the  study  (Krathwohl, 
1998).  The  treatment  in  this  study  was  the  respondents  selecting  the  maturity  elements 
in  the  framework  as  adding  value.  However,  the  respondents  were  not  restricted  to 
select  any  or  all  of  the  cards  if  they  did  not  feel  they  were  of  value.  Furthermore, 
respondents  were  allowed  to  create  maturity  elements  not  on  the  cards  allowing  them 
to  offer  what  they  perceived  as  adding  value  regardless  if  the  maturity  element  was  on 
cards. 

Observations  and  Measures. 

This  facet  deals  with  how  representative  the  chosen  measures  and  instruments 
used  during  the  study  are  of  all  possible  measures  and  instruments  for  similar  studies 
(Krathwohl,  1998).  The  cards  used  during  the  exercises  were  the  measurement  tools 
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used  to  gather  the  quantitative  data  and  focus  the  qualitative  discussion.  These  cards 
do  not  capture  all  possible  maturity  elements  that  a  respondent  could  value  during 
early  development,  but  for  the  purposes  of  validating  the  framework  the  focus  used  is 
appropriate.  The  research  team  did  include  a  few  extraneous  cards  to  the  framework 
to  increase  the  possible  responses  respondents  could  give. 

Basis  for  Sensing  Attributes  or  Changes  Link  and  Procedure  Link. 

Again,  Krathwohl  (1998)  combines  these  two  design  links  for  this  facet  of 
translation  of  fidelity.  According  to  Krathwohl,  these  design  links  should  be 
representative  of  similar  studies  and  that  unusual  or  overly  restrictive  methods  to  sense 
changes  or  conduct  the  procedures  degrade  external  validity.  Asking  the  respondents 
to  comment  on  the  cards  regardless  of  the  individual  and  then  recording  their 
comments  is  common  to  any  market  research  related  to  conjoint  analysis. 

Furthermore,  the  questions  asked  and  procedures  of  the  study  could  be  conducted 
anywhere  and  any  researcher  with  a  basic  knowledge  of  early  development  in  the  DoD 
could  administer  the  interviews. 

Time. 

The  final  facet  for  translation  fidelity  is  time,  which  though  different  from  a 
design  link  for  internal  validity,  Krathwohl  (1998)  claims  is  a  very  important  factor  to 
infer  generality.  Time  is  important  as  any  generalizations  regarding  findings  within  a 
study,  decay  with  the  data  and  information  used  during  the  study.  Many  of  the 
respondents  had  experience  ranging  from  a  couple  years  to  over  30  years  in  DoD  and 
industry  acquisitions  and  development  with  varying  experience  in  the  early  stages. 
However,  to  focus  the  responses  based  on  the  current  or  recent  perceptions  and  to 
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reduce  the  likelihood  that  these  responses  are  irrelevant  or  outdated,  the  research  team 
asked  each  respondent  to  focus  their  responses  on  current  experiences  and  up  to  five 
years  prior.  Additionally,  the  primary  findings  on  what  generic  information  is  of  value 
to  a  decision  maker  should  be  relevant  regardless  of  the  timeframe  used. 

“Demonstrated  Generality” 

Next,  is  “demonstrated  generality”,  which  “is  a  judgment  of  the  extent  to 
which  the  relationship  appeared  in  all  instances  of  the  study  in  which  it  would  be 
expected  to  do  so  and  did  not  where  it  shouldn’t”  (Krathwohl,  1998,  p.l80). 

Krathwohl  purposefully  places  quotations  around  this  judgment  to  emphasize,  “the 
logical  impossibility  of  demonstrating  generality  in  all  the  instances  where  it  is 
intended  to  apply”  (p.l80).  However,  the  research  team  offers  that  both  the 
application  of  the  framework  and  the  confirmatory  sources  discussed  previously  in  the 
methodology  and  analysis  and  results  chapters  help  demonstrate  the  generality  of  the 
findings  from  the  interviews. 

The  application  shows  how  even  from  an  academic  point  of  view  most  of  the 
maturity  elements  can  be  at  least  started  very  early  on  in  concept  development.  The 
researchers  were  able  to  construct  many  of  the  maturity  elements  for  the  sponsoring 
organization,  who  indicated  that  these  deliverables  were  helpful  to  understanding  the 
concept.  Additionally,  the  sponsoring  organization  was  not  an  Air  Force  organization, 
but  rather  a  separate  DoD  agency,  which  demonstrates  that  the  framework  can  be 
applied  to  organizations  external  to  the  interview  study. 

Although  the  discussion  on  the  confirmatory  sources  of  the  framework  are 
neither  mean  to  infer  that  the  framework  adds  value  nor  that  it  should  be 
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recommended  for  use,  they  do  help  support  the  findings  of  this  study’s  generality.  The 
confirmatory  sources  demonstrate  that  specific  maturity  elements  within  the 
framework  are  accepted  by  DoD  policy.  More  importantly,  Air  Force  guidance  that  is 
founded  on  inputs  from  practitioners,  experts  in  the  field,  and  best-practices  (i.e.  the 
CCTD  and  Early  SE  guides)  also  supports  use  of  elements  within  the  framework.  In 
summary,  the  research  team  is  confident  that  the  demonstrated  generality  of  this 
study’s  findings  are  favorable  to  reasonably  pass  this  third  judgment. 

Restrictive  Explanations  Eliminated 

Krathwohl’s  fourth  judgment  for  external  validity  is  restrictive  explanations 
eliminated.  Krathwohl  (1998)  describes  this  judgment  as,  “restrictive  explanations 
(conditions)  that  were  part  of  the  study  but  would  not  be  part  of  the  target  of 
generalization  must  have  been  eliminated  for  the  inferential  leap  to  the  target  to  be 
confidently  made”  (p.  181).  He  explains  further  that  if  the  target  the  researcher  is 
trying  to  generalize  the  findings  towards  is  not  actually  represented  in  the  study;  then 
the  researcher  must  discuss  why  reasons  that  would  otherwise  restrict  using  the  target 
in  the  study  should  be  disregarded. 

In  terms  of  this  validation  effort,  the  research  team  is  seeking  to  generalize  the 
findings  of  the  interviews  past  the  individuals  interviewed  and  towards  the  target  of 
the  greater  Air  Eorce  and  related  DoD  organizations.  As  discussed  in  the  previous 
judgments,  the  researchers  do  not  see  any  reasons  that  would  restrict  the  interview 
results  for  additional  individuals  in  similar  organizations.  Each  interview  was 
conducted  privately  and  allowed  for  the  individual  to  respond  based  solely  on  their 
opinion  without  outside  bias  and  were  not  restricted  in  forming  any  of  their  responses. 
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The  same  interview  approach  could  be  used  for  further  studies  with  a  different  sample 
group.  The  researchers  have  no  preconception  that  the  results  will  be  the  same  as  in 
this  study,  but  only  offer  that  nothing  should  restrict  the  same  result  from  occurring. 

Replicable  Result 

The  final  judgment  for  external  validity  as  discussed  by  Krathwohl  (1998)  is  a 
replicable  result,  which  “is  a  summary  judgment  of  the  forgoing  judgments  and  of  the 
extent  to  which  the  results  of  this  study  could  be  replicated  in  the  target  to  which  it  is 
being  generalized”  (p.l81).  This  judgment  is  similar  to  its  counterpart  for  internal 
validity,  a  credible  result,  and  essentially  synthesizes  the  other  judgments  for  a 
conclusion  on  the  strength  of  external  validity  (Krathwohl,  1998).  Krathwohl  states 
that  this  judgment  is  at  the  heart  of  external  validity  and  asks  would  similar  results 
reproduce  in  the  target  of  generalization.  The  research  team  believes  that  if  the  same 
type  of  study  were  conducted  with  a  different  sample  that  similar  results  regarding  the 
framework’s  value  would  be  reached.  The  research  team  is  confident  that  the 
discussion  on  the  previous  four  judgments  can  lead  one  to  accept  this  study’s 
generality  to  the  Air  Force  and  other  DoD  military  branches  and  agencies  with  fair  to 
strong  confidence. 
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